Pinpoint QuickStart

얼마 전에 Hadoop, Hbase 최근 버전을 설치한 후, 네이버 Pinpint 설치를 시도해 봤습니다. 하지만 Pinpoint가 Hadoop 최신 버전을 공식 지원하지 않아서인지 실패했습니다.

어쨌든 빨리 사용해 보고 싶어서 QuickStart 가이드를 따라서 설치해 봅니다.

Linux 환경에서 QuickStart 실행하기

Pinpoint QuickStart 가이드 문서는 아래 주소에서 볼 수 있습니다(영문).

https://github.com/naver/pinpoint/tree/master/quickstart

리눅스 환경에서의 설치를 가이드하고 있어서 저는 AWS에 설치한 CentOS 7에 설치했습니다.

아래 주소에 가면 윈도 환경에서 실행하는 방법을 볼 수 있습니다.
Windows 환경에서 QuickStart 실행하기

Install JDK 6

Pinpoint 실행을 위해서는 JDK 6와 JDK 7+, Maven 3.2x+ 버전을 먼저 설치해야 합니다.

우선 JDK 설치를 위해서 wget 부터 설치합니다.

[centos@ip-172-31-7-87 ~]$ sudo -i
[root@ip-172-31-7-87 ~]# yum install -y wget

wget으로 JDK 6 바이너리(jdk-6u45-linux-x64.bin)를 다운로드 받은 후, chmod 명령으로 실행권한을 부여한 후 설치합니다.

[root@ip-172-31-7-87 ~]# wget http://ghaffarian.net/downloads/Java/JDK/jdk-6u45-linux-x64.bin
--2015-03-25 11:03:07--  http://ghaffarian.net/downloads/Java/JDK/jdk-6u45-linux-x64.bin
Resolving ghaffarian.net (ghaffarian.net)... 178.32.91.18
Connecting to ghaffarian.net (ghaffarian.net)|178.32.91.18|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 72087592 (69M) [application/octet-stream]
Saving to: ‘jdk-6u45-linux-x64.bin’

100%[=================================================>] 72,087,592  5.52MB/s   in 25s

2015-03-25 11:03:33 (2.78 MB/s) - ‘jdk-6u45-linux-x64.bin’ saved [72087592/72087592]

[root@ip-172-31-7-87 ~]# 
[root@ip-172-31-7-87 ~]# chmod u+x jdk-6u45-linux-x64.bin
[root@ip-172-31-7-87 ~]# ./jdk-6u45-linux-x64.bin
Unpacking...
Checksumming...
Extracting...
UnZipSFX 5.50 of 17 February 2002, by Info-ZIP (Zip-Bugs@lists.wku.edu).
   creating: jdk1.6.0_45/
   creating: jdk1.6.0_45/db/
   creating: jdk1.6.0_45/db/bin/
  inflating: jdk1.6.0_45/db/bin/ij
  inflating: jdk1.6.0_45/db/bin/NetworkServerControl
  inflating: jdk1.6.0_45/db/bin/setNetworkClientCP.bat
  inflating: jdk1.6.0_45/db/bin/setEmbeddedCP.bat
....
....
....
  inflating: jdk1.6.0_45/include/classfile_constants.h
  inflating: jdk1.6.0_45/COPYRIGHT
Creating jdk1.6.0_45/jre/lib/rt.jar
Creating jdk1.6.0_45/jre/lib/jsse.jar
Creating jdk1.6.0_45/jre/lib/charsets.jar
Creating jdk1.6.0_45/lib/tools.jar
Creating jdk1.6.0_45/jre/lib/ext/localedata.jar
Creating jdk1.6.0_45/jre/lib/plugin.jar
Creating jdk1.6.0_45/jre/lib/javaws.jar
Creating jdk1.6.0_45/jre/lib/deploy.jar

Done.

설치 후 생성된 jdk1.6.0_45 디렉토리를 /opt 디렉토리로 이동합니다.

[root@ip-172-31-7-87 ~]# mv jdk1.6.0_45 /opt

Install JDK 8

wget으로 JDK 8 압축 파일(jdk-8u25-linux-x64.tar.gz)을 다운로드 받은 후, /opt에 압축을 해제합니다.

압축 해제가 완료되면 /opt/jdk1.8.0_25 디렉토리가 생성됩니다.

[root@ip-172-31-7-87 ~]# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u25-b17/jdk-8u25-linux-x64.tar.gz"

[root@ip-172-31-7-87 ~]# tar -zxvf jdk-8u25-linux-x64.tar.gz -C /opt

alternatives 명령을 사용하여 어디에서나 java 명령을 사용할 수 있도록 설정합니다.

[root@ip-172-31-7-87 ~]#  /usr/sbin/alternatives --install /usr/bin/java java /opt/jdk1.8.0_25/bin/java 100

[root@ip-172-31-7-87 ~]# java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

/etc/profile 파일을 편집하여 JAVA_HOME, JAVA_7_HOME, JAVA_6_HOME 환경변수를 설정합니다.

JAVA_HOME, JAVA_7_HOME은 앞에서 설치한 JDK 8을 가리키도록 설정했습니다.

[root@ip-172-31-7-87 ~]# vi /etc/profile
export JAVA_HOME=/opt/jdk1.8.0_25
export JAVA_7_HOME=/opt/jdk1.8.0_25
export JAVA_6_HOME=/opt/jdk1.6.0_45

source 명령으로 방금 편집한 /etc/profile을 새로 읽어 들인 후, 환경변수 설정이 잘 됐는지 확인합니다.

[root@ip-172-31-7-87 ~]# source /etc/profile

[root@ip-172-31-7-87 ~]# echo $JAVA_7_HOME
/opt/jdk1.8.0_25

[root@ip-172-31-7-87 ~]# echo $JAVA_6_HOME
/opt/jdk1.6.0_45

Install Maven

지금까지는 JDK 설치를 위해서 root 계정을 사용했습니다만, Maven은 centos 계정으로 설치합니다. Pinpoint 실행 또한 centos 계정을 사용합니다.

Maven 3.2.5 버전을 wget으로 다운로드 받은 후 압축을 해제합니다.

[root@ip-172-31-7-87 ~]# exit
[centos@ip-172-31-7-87 ~]# source /etc/profile
[centos@ip-172-31-7-87 ~]# wget http://apache.tt.co.kr/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
--2015-03-25 11:16:07--  http://apache.tt.co.kr/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
Resolving apache.tt.co.kr (apache.tt.co.kr)... 121.125.79.185
Connecting to apache.tt.co.kr (apache.tt.co.kr)|121.125.79.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7956528 (7.6M) [application/x-gzip]
Saving to: ‘apache-maven-3.2.5-bin.tar.gz’

100%[======================================>] 7,956,528    124KB/s   in 62s

2015-03-25 11:17:09 (125 KB/s) - ‘apache-maven-3.2.5-bin.tar.gz’ saved [7956528/7956528]

[centos@ip-172-31-7-87 ~]$ tar xfz apache-maven-3.2.5-bin.tar.gz

.bashrc 파일을 편집하여 MAVEN_HOME, PATH 환경변수를 추가합니다.

[centos@ip-172-31-7-87 ~]$ vi ~/.bashrc
export MAVEN_HOME=/home/centos/apache-maven-3.2.5
export PATH=$PATH:$MAVEN_HOME/bin

mvn -version 명령을 실행하여 설치가 잘 됐는지 확인합니다. mvn은 Maven 실행파일입니다.

[centos@ip-172-31-7-87 ~]$ mvn -version
mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /home/centos/apache-maven-3.2.5
Java version: 1.8.0_25, vendor: Oracle Corporation
Java home: /opt/jdk1.8.0_25/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-123.8.1.el7.x86_64", arch: "amd64", family: "unix"

Pinpoint 저장소 복제

Pinpoint 저장소를 복제(clone)하기 위해서 root 계정으로 Git을 설치합니다.

[centos@ip-172-31-7-87 ~]$ sudo -i
[root@ip-172-31-7-87 ~]# yum install -y git
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.riken.jp
 * extras: ftp.riken.jp
 * updates: ftp.riken.jp
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.8.3.1-4.el7 will be installed
--> Processing Dependency: perl-Git = 1.8.3.1-4.el7 for package: git-1.8.3.1-4.el7.x86_64
--> Processing Dependency: perl >= 5.008 for package: git-1.8.3.1-4.el7.x86_64
....
....
....
  perl-Time-Local.noarch 0:1.2300-2.el7    perl-constant.noarch 0:1.27-2.el7
  perl-libs.x86_64 4:5.16.3-283.el7        perl-macros.x86_64 4:5.16.3-283.el7
  perl-parent.noarch 1:0.225-244.el7       perl-podlators.noarch 0:2.5.1-3.el7
  perl-threads.x86_64 0:1.87-4.el7         perl-threads-shared.x86_64 0:1.43-6.el7

Complete!

git clone 명령으로 Pinpoint 저장소를 복제합니다.

[root@ip-172-31-7-87 ~]# exit
logout

[centos@ip-172-31-7-87 ~]$ git clone https://github.com/naver/pinpoint.git
Cloning into 'pinpoint'...
remote: Counting objects: 93939, done.
remote: Compressing objects: 100% (218/218), done.
remote: Total 93939 (delta 111), reused 0 (delta 0), pack-reused 93579
Receiving objects: 100% (93939/93939), 151.77 MiB | 7.53 MiB/s, done.
Resolving deltas: 100% (48776/48776), done.

mvn install -Dmaven.test.skip=true 명령으로 Pinpoint를 설치합니다.

maven.test.skip=true은 단위테스트를 건너뛰기 위한 옵션입니다

[centos@ip-172-31-7-87 ~]$ cd pinpoint
[centos@ip-172-31-7-87 pinpoint]$ mvn install -Dmaven.test.skip=true
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] pinpoint
[INFO] pinpoint-thrift
[INFO] pinpoint-commons
[INFO] pinpoint-bootstrap-core
[INFO] pinpoint-bootstrap
[INFO] pinpoint-rpc
[INFO] pinpoint-profiler
[INFO] pinpoint-profiler-optional
[INFO] pinpoint-jdk-http-plugin
[INFO] pinpoint-test
[INFO] pinpoint-redis-plugin
[INFO] pinpoint-servlet-plugin
[INFO] pinpoint-tomcat-plugin
[INFO] pinpoint-plugins
[INFO] pinpoint-agent-distribution
[INFO] pinpoint-collector
[INFO] pinpoint-web
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building pinpoint 1.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: http://repository.codehaus.org/org/apache/maven/plugins/maven-install-plugin/2.4/maven-install-plugin-2.4.pom
....
....
....
[INFO] Installing /home/centos/pinpoint/web/target/pinpoint-web-1.1.0-SNAPSHOT-classes.jar to /home/centos/.m2/repository/com/navercorp/pinpoint/pinpoint-web/1.1.0-SNAPSHOT/pinpoint-web-1.1.0-SNAPSHOT-classes.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] pinpoint ........................................... SUCCESS [ 23.635 s]
[INFO] pinpoint-thrift .................................... SUCCESS [01:51 min]
[INFO] pinpoint-commons ................................... SUCCESS [01:56 min]
[INFO] pinpoint-bootstrap-core ............................ SUCCESS [  1.393 s]
[INFO] pinpoint-bootstrap ................................. SUCCESS [ 21.062 s]
[INFO] pinpoint-rpc ....................................... SUCCESS [  5.175 s]
[INFO] pinpoint-profiler .................................. SUCCESS [ 48.396 s]
[INFO] pinpoint-profiler-optional ......................... SUCCESS [  0.910 s]
[INFO] pinpoint-jdk-http-plugin ........................... SUCCESS [  0.990 s]
[INFO] pinpoint-test ...................................... SUCCESS [ 34.903 s]
[INFO] pinpoint-redis-plugin .............................. SUCCESS [  1.074 s]
[INFO] pinpoint-servlet-plugin ............................ SUCCESS [  0.887 s]
[INFO] pinpoint-tomcat-plugin ............................. SUCCESS [  1.096 s]
[INFO] pinpoint-plugins ................................... SUCCESS [  0.007 s]
[INFO] pinpoint-agent-distribution ........................ SUCCESS [ 55.523 s]
[INFO] pinpoint-collector ................................. SUCCESS [ 50.376 s]
[INFO] pinpoint-web ....................................... SUCCESS [ 42.698 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:37 min
[INFO] Finished at: 2015-03-25T11:38:15+00:00
[INFO] Final Memory: 33M/139M
[INFO] ------------------------------------------------------------------------

HBase 설치, 시작, 초기화

~/pinpoint/quickstart/bin/start-hbase.sh 스크립트를 이용해서 hbase를 다운로드하고 시작합니다.

[centos@ip-172-31-7-87 pinpoint]$ quickstart/bin/start-hbase.sh
Hbase not detected.
Downloading hbase...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 56.5M  100 56.5M    0     0  4030k      0  0:00:14  0:00:14 --:--:-- 5817k
starting master, logging to /home/centos/pinpoint/quickstart/hbase/hbase-0.94.25/bin/../logs/hbase-centos-master-ip-172-31-7-87.out

~/pinpoint/quickstart/bin/init-hbase.sh 스크립트를 이용해서 hbase 테이블을 초기화(initialize)합니다.

[centos@ip-172-31-7-87 pinpoint]$ quickstart/bin/init-hbase.sh
0 row(s) in 1.7520 seconds

0 row(s) in 1.0980 seconds

0 row(s) in 1.0580 seconds

0 row(s) in 2.7240 seconds

0 row(s) in 1.0700 seconds

0 row(s) in 1.0770 seconds

0 row(s) in 2.2110 seconds

0 row(s) in 1.0990 seconds

0 row(s) in 1.0650 seconds

0 row(s) in 1.0530 seconds

0 row(s) in 1.0590 seconds

0 row(s) in 1.0660 seconds

0 row(s) in 1.0470 seconds

0 row(s) in 1.0570 seconds

TABLE
AgentInfo
AgentStat
ApiMetaData
ApplicationIndex
ApplicationMapStatisticsCallee
ApplicationMapStatisticsCaller
ApplicationMapStatisticsSelf
ApplicationStatistics
ApplicationTraceIndex
HostApplicationMap
HostApplicationMap_Ver2
SqlMetaData
StringMetaData
Traces
14 row(s) in 0.0340 seconds

Collector 시작

~/pinpoint/quickstart/bin/start-collector.sh 스크립트를 이용해서 Pinpoint collector를 시작합니다.

[centos@ip-172-31-7-87 pinpoint]$ quickstart/bin/start-collector.sh
---check pinpoint-quickstart-collector process status.---
---initialize pinpoint-quickstart-collector logs.---
mkdir /home/centos/pinpoint/quickstart/logs
mkdir /home/centos/pinpoint/quickstart/logs/pid
---pinpoint-quickstart-collector initialization started. pid=10526.---
starting pinpoint-quickstart-collector. 0 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 5 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 10 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 15 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 20 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 25 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 30 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 35 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 40 /180 sec(close wait limit).
starting pinpoint-quickstart-collector. 45 /180 sec(close wait limit).
---pinpoint-quickstart-collector initialization completed. pid=10526.---
03-25 11:41:54 [INFO ](c.n.p.c.r.u.BaseUDPReceiver        :230) Pinpoint-UDP-Stat start.
03-25 11:41:54 [WARN ](c.n.p.c.r.u.BaseUDPReceiver        :211) DatagramSocket.setReceiveBu fferSize() error. 4194304!=212992
03-25 11:41:54 [INFO ](c.n.p.c.r.u.BaseUDPReceiver        :236) UDP Packet reader:2 started .
03-25 11:41:54 [INFO ](c.n.p.c.r.u.BaseUDPReceiver        :132) start ioThread localAddress :0.0.0.0/0.0.0.0, IoThread:Pinpoint-UDP-Span-Io(11-0)
03-25 11:41:54 [INFO ](c.n.p.c.r.u.BaseUDPReceiver        :132) start ioThread localAddress :0.0.0.0/0.0.0.0, IoThread:Pinpoint-UDP-Stat-Io(13-0)
03-25 11:41:54 [INFO ](c.n.p.c.r.u.BaseUDPReceiver        :132) start ioThread localAddress :0.0.0.0/0.0.0.0, IoThread:Pinpoint-UDP-Span-Io(11-1)
03-25 11:41:54 [INFO ](c.n.p.c.r.u.BaseUDPReceiver        :132) start ioThread localAddress :0.0.0.0/0.0.0.0, IoThread:Pinpoint-UDP-Stat-Io(13-1)
03-25 11:41:54 [INFO ](o.s.w.c.ContextLoader              :313) Root WebApplicationContext:  initialization completed in 2246 ms
Mar 25, 2015 11:41:54 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-28082"]

Web UI 시작

~/pinpoint/quickstart/bin/start-web.sh 스크립트를 이용해서 Pinpoint Web UI를 시작합니다.

[centos@ip-172-31-7-87 pinpoint]$ quickstart/bin/start-web.sh
---check pinpoint-quickstart-web process status.---
---initialize pinpoint-quickstart-web logs.---
---pinpoint-quickstart-web initialization started. pid=13397.---
starting pinpoint-quickstart-web. 0 /180 sec(close wait limit).
starting pinpoint-quickstart-web. 5 /180 sec(close wait limit).
starting pinpoint-quickstart-web. 10 /180 sec(close wait limit).
starting pinpoint-quickstart-web. 15 /180 sec(close wait limit).
starting pinpoint-quickstart-web. 20 /180 sec(close wait limit).
starting pinpoint-quickstart-web. 25 /180 sec(close wait limit).
---pinpoint-quickstart-web initialization completed. pid=13397.---
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/transactionInfo/] onto handler 'businessTransactionController'
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/transactionList] onto handler 'businessTransactionController'
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/transactionList.*] onto handler 'businessTransactionController'
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/transactionList/] onto handler 'businessTransactionController'
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/lastTransactionList] onto handler 'businessTransactionController'
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/lastTransactionList.*] onto handler 'businessTransactionController'
11:44:08 INFO (m.a.DefaultAnnotationHandlerMapping:315) Mapped URL path [/lastTransactionList/] onto handler 'businessTransactionController'
11:44:08 INFO (o.s.w.s.DispatcherServlet          :473) FrameworkServlet 'pinpoint-web': initialization completed in 1265 ms
Mar 25, 2015 11:44:08 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-28080"]

TestApp 시작

~/pinpoint/quickstart/bin/start-testapp.sh 스크립트를 이용해서 Pinpoint TestApp을 시작합니다.

[centos@ip-172-31-7-87 ~]$ quickstart/bin/start-testapp.sh
-bash: quickstart/bin/start-testapp.sh: No such file or directory
[centos@ip-172-31-7-87 ~]$ cd pin
-bash: cd: pin: No such file or directory
[centos@ip-172-31-7-87 ~]$ cd pinpoint/
[centos@ip-172-31-7-87 pinpoint]$ quickstart/bin/start-testapp.sh
---check pinpoint-quickstart-testapp process status.---
---initialize pinpoint-quickstart-testapp logs.---
---initialize pinpoint-quickstart-testapp agent.---
---pinpoint-quickstart-testapp initialization started. pid=4048.---
starting pinpoint-quickstart-testapp. 0 /180 sec(close wait limit).
starting pinpoint-quickstart-testapp. 5 /180 sec(close wait limit).
starting pinpoint-quickstart-testapp. 10 /180 sec(close wait limit).
starting pinpoint-quickstart-testapp. 15 /180 sec(close wait limit).
---pinpoint-quickstart-testapp initialization completed. pid=4048.---
2015-03-25 11:48:29 [DEBUG](c.n.p.p.i.b.AspectWeaverClass      :149) JointPoint method __getHeaders(Ljava/lang/String;)Ljava/util/Enumeration; -> invokeOriginal:$_=__getHeaders_$$pinpoint($$);
....
....
....
2015-03-25 11:49:05 [WARN ](c.n.p.r.c.PinpointClientSocketHandshaker:260) Handshake already completed.

확인

설치 및 실행이 완료되면 아래 주소에서 Pinpoint Web UI와 TestApp의 동작을 확인할 수 있습니다.

Web UI - http://localhost:28080

TestApp - http://localhost:28081

TestApp에 있는 여러 링크를 눌러본 후, Pinpoint가 어떻게 동작하는지 확인합니다.

종료

각각 아래의 스크립트로 종료합니다.

HBase - ~/pinpoint/quickstart/bin/stop-hbase.sh

Collector ~/pinpoint/quickstart/bin/stop-collector.sh

Web UI - ~/pinpoint/quickstart/bin/stop-web.sh

TestApp - ~/pinpoint/quickstart/bin/stop-testapp.sh

WSL 2(Windows Subsystem for Linux 2) 설치 (0)	2020.12.13
Ubuntu 16.04에서 발생하는 libseccomp2 의존성 문제 (2)	2019.12.23
CentOS 7에 Hadoop 설치 (2)	2015.03.09
CentOS 공개키 인증으로 암호 없이 로그인 (0)	2014.10.18
시스템 환경변수 설정 (0)	2007.10.28

Pinpoint 빨리 설치하기(QuickStart)

Pinpoint QuickStart

Linux 환경에서 QuickStart 실행하기

Install JDK 6

Install JDK 8

Install Maven

Pinpoint 저장소 복제

HBase 설치, 시작, 초기화

Collector 시작

Web UI 시작

TestApp 시작

확인

종료

'설치&설정' 카테고리의 다른 글

최근 글

최근 월별 글

최근 댓글

최근 글

최근 월별 글

최근 댓글

최근 글

최근 월별 글

최근 댓글

지난달

2024.11

다음달

티스토리툴바

일	월	화	수	목	금	토
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30