Oracle Linux/Common Tools
(→Testing) |
|||
| Line 163: | Line 163: | ||
==Testing== | ==Testing== | ||
| − | <nowiki>Storage Device Burn-In | + | <nowiki>Storage Device Burn-In |
Spinning disk hard drives have moving parts, by definition. These parts are highly-sensitive to shock and vibration, and will eventually wear out with use. Consider pre-flighting every storage device before putting it into production, paying attention to: | Spinning disk hard drives have moving parts, by definition. These parts are highly-sensitive to shock and vibration, and will eventually wear out with use. Consider pre-flighting every storage device before putting it into production, paying attention to: | ||
Revision as of 16:20, 29 June 2021
Common Tools
Contents |
Listening Ports
netstat
~$ netstat -lnp Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:17003 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:40333 0.0.0.0:* LISTEN - udp 42240 0 127.0.0.1:8125 0.0.0.0:* - udp 32256 0 169.254.0.2:53 0.0.0.0:* -
Determine process listening port(s) by name
~$ sudo netstat -tupln Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 374/apache2
determine the location of the process using the PID
~$ sudo ls -la /proc/374/exe lrwxrwxrwx 1 root root 0 2020-01-27 00:38 /proc/374/exe -> /usr/lib/apache2/mpm-prefork/apache2
process activity
top
~$ top
LoadAvg Load average over 1, 5, and 15 minutes. divide by number of cores to determine load.
ps
ps -AlH
This shows all the processes running, in long format, with the extra full format output
Network Throughput
iperf
requires a server/client setup that communicates over port 5001. generally server/client placement is dependent on network accessibility, ie. server would be placed outside of the network, client within. You may need to open 5001 on both sides however, in case established/related is not implicitly accepted from the client side > out.
Server side
~$ sudo yum install -y iperf sudo iptables -A INPUT -p tcp --dport 5001 -j ACCEPT -m comment --comment "iperf - 5001" ~$ iperf -s
Example:
$ iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.1 port 5001 connected with 10.0.0.2 port 47552 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 76.6 MBytes 64.1 Mbits/sec [ 5] local 10.0.0.1 port 5001 connected with 10.0.0.2 port 39914 [ 5] 0.0-10.0 sec 84.5 MBytes 70.6 Mbits/sec
Client side
~$ iperf -c target.ip -p port.number
- Example
$ iperf -c 10.0.0.1 -p 5001 ------------------------------------------------------------ Client connecting to 10.0.0.1, TCP port 5001 TCP window size: 466 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.1 port 39914 connected with 10.0.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 84.5 MBytes 70.8 Mbits/sec
Disk
lsblk
Shows a list of block devices. Useful in showing mount points, size, name, etc.
~$ lsblk --all NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 0 loop loop1 7:1 0 0 loop loop2 7:2 0 0 loop loop3 7:3 0 0 loop loop4 7:4 0 0 loop loop5 7:5 0 0 loop loop6 7:6 0 0 loop loop7 7:7 0 0 loop sr0 11:0 1 1024M 0 rom sda 8:0 0 8G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 7.5G 0 part ├─VolGroup-lv_root (dm-0) 253:0 0 6.7G 0 lvm / └─VolGroup-lv_swap (dm-1) 253:1 0 816M 0 lvm [SWAP]
fdisk
Tool for showing partitions and disks, as well as other things
~$ sudo fdisk -l Disk /dev/sda: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000091fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 1045 7875584 8e Linux LVM Disk /dev/mapper/VolGroup-lv_root: 7205 MB, 7205814272 bytes 255 heads, 63 sectors/track, 876 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/VolGroup-lv_swap: 855 MB, 855638016 bytes 255 heads, 63 sectors/track, 104 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
df
shows the amount of disk space on the file system and the file system type. Can also show inodes.
~$ df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/centos-root xfs 6.2G 1.5G 4.8G 24% / devtmpfs devtmpfs 485M 0 485M 0% /dev tmpfs tmpfs 496M 0 496M 0% /dev/shm tmpfs tmpfs 496M 6.8M 490M 2% /run tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/sda1 xfs 1014M 159M 856M 16% /boot tmpfs tmpfs 100M 0 100M 0% /run/user/1000
du
shows the amount of space used from a specific directory and recursively includes its children.
~$ du -h /home/user/temp 4.0K /home/user/temp
Find the largest 20 files/folders on a system
$ sudo du -a / 2>/dev/null | sort -n -r | head -n 20 1893852 / 990024 /var 447452 /home 447448 /home/r00t 429228 /home/r00t/backup 429224 /home/r00t/backup/4_2_2019 423912 /usr 421432 /var/www 421420 /var/www/mediawiki-1.19.2 362128 /home/r00t/backup/4_2_2019/wikidb4_2_2019.tgz 277372 /var/www/mediawiki-1.19.2/images 273504 /var/www/mediawiki-1.19.2/images/upload 251020 /var/lib 223472 /var/cache 219792 /var/cache/apt 202204 /usr/share 188916 /var/cache/apt/archives 171208 /var/lib/mysql 157700 /var/lib/mysql/ibdata1 128896 /var/www/mediawiki-1.19.2/images/upload/20140222_162517.mp4
local mounts and file systems
~$ sudo cat /proc/mounts rootfs / rootfs rw 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,seclabel,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,relatime,size=499208k,nr_inodes=124802,mode=755 0 0 devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /dev/shm tmpfs rw,seclabel,relatime 0 0 /dev/mapper/VolGroup-lv_root / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0 none /selinux selinuxfs rw,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,relatime,size=499208k,nr_inodes=124802,mode=755 0 0 /proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0 /dev/sda1 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
Testing
Storage Device Burn-In Spinning disk hard drives have moving parts, by definition. These parts are highly-sensitive to shock and vibration, and will eventually wear out with use. Consider pre-flighting every storage device before putting it into production, paying attention to: Start a long HDD self test (smartctl -t long /dev/) After the test is done (could take 12+ hours), check the results (smartctl -a /dev/) Pending sector reallocations (smartctl -a /dev/ | grep Current_Pending_Sector) Reallocated sector count (smartctl -a /dev/ | grep Reallocated_Sector_Ct) UDMA CRC errors (smartctl -a /dev/ | grep UDMA_CRC_Error_Count) HDD and SSD write latency consistency (diskinfo -wS ) Unformatted drives only! HDD and SSD hours (smartctl -a /dev/ | grep Power_On_Hours) NVMe percentage used (nvmecontrol logpage -p 2 nvme0 | grep “Percentage used”) Take time before deploying the system to create a pool and subject it to as close to the real-world workload as possible. This can reveal individual drive issues but also can help determine if an alternative pool layout is better suited to that workload. Be cautious of used drives as vendors may not be honest or informed about the age and health of any given drive. Check the number of hours on all “new” drives using smartctl(8) as they may in fact be “recertified” or simply untruthfully advertised. A drive vendor may also zero the hours of a drive during recertification, masking its true age. All storage devices sold by iXsystems are tested for a minimum of 48 hours prior to shipment.
hardware/driver
dmesg
Used to gather information about hardware and drivers on a system but can also be used to troubleshoot and show verbose logs pertaining to such.
$ dmesg | less [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.11.0-13-generic (buildd@aatxe) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #20-Ubuntu SMP Wed Oct 23 17:26:33 UTC 2013 (Ubuntu 3.11.0-13.20-generic 3.11.6) [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] NSC Geode by NSC [ 0.000000] Cyrix CyrixInstead [ 0.000000] Centaur CentaurHauls [ 0.000000] Transmeta GenuineTMx86 [ 0.000000] Transmeta TransmetaCPU [ 0.000000] UMC UMC UMC UMC [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007dc08bff] usable
Find specific device
~$ dmesg | grep sda [ 1.280971] sd 2:0:0:0: [sda] 488281250 512-byte logical blocks: (250 GB/232 GiB) [ 1.281014] sd 2:0:0:0: [sda] Write Protect is off [ 1.281016] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 1.281039] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.359585] sda: sda1 sda2 < sda5 sda6 sda7 sda8 > [ 1.360052] sd 2:0:0:0: [sda] Attached SCSI disk
Downloading
wget
quiet
-q | --quiet- Turn off Wget's output
verbose
-v | --verbose- Turn on verbose output
input file
-i | --input-file=file- Reads URLs from a local or external file.
Tries
-t | --tries=number- number of times to retry.
Output
-O | --output-document=file- all documents will be concatenated together and written to the file
Progress Indicator
--progress=type- Show a process indicator with varying types, such as dot and bar
CLI
telnet
example: HTTP request
Below is a manual telnet example:
~$ telnet stackoverflow.com 80 Trying 151.101.65.69... Connected to stackoverflow.com. Escape character is '^]'. GET /questions HTTP/1.0 Host: stackoverflow.com HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 ...
find
Find the largest files in a directory
$ find $HOME -type f -printf '%s %p\n' | sort -nr | head -10 370813860 /home/r00t/backup/4_2_2019/wikidb4_2_2019.tgz 68699129 /home/r00t/backup/4_2_2019/wikidb4_2_2019.sql 18266096 /home/r00t/downloads/mediawiki-1.19.2.tar.gz 55320 /home/r00t/downloads/Vector-MW1.19-112648.tar.gz 40273 /home/r00t/.bash_history 26023 /home/r00t/temp/website/assistfuncs.js 22139 /home/r00t/Banned_IPs.txt 14919 /home/r00t/temp/website/image_db.html 11409 /home/r00t/.viminfo 10273 /home/r00t/temp/website/softwareupdate.html
ulimit
determine max number of processes per user.
~$ ulimit -a | grep processes max user processes (-u) unlimited
temp
http get
parameters are passed inline with the URL request.
http://test.com?param=foo,param=bar
http post
parameters are passed within the body
~$ telnet test.com 80 POST /index.html HTTP/1.1 Host: test.com param=foo, param=bar
javascript sop
- Same-Origin Policy (SOP)
common when using javascript to integrated with external resources, like an API.
there are security measures within browsers that restrict interaction between documents (or scripts) that have different origins.
decrypt ssl
client
use pre-master key captured by the client side using an environment variable populated when a browser visits a SSL site. Afterwards used in wireshark to decrypt SSL traffic sent to/from the client.
server
from the server side it can be a bit more complicated depending on the type of encryption.
SSL decryption works only if the key exchange was of type RSA or static DH; with "DHE" and "ECDHE" cipher suites, you won't be able to decrypt such a session, even with knowledge of the server private key. In that case, you will need either the negotiated "master secret", or to use the server private key to actively intercept the connection (in a Man-in-the-Middle setup).
Akamai (CDN)
Akamai is a CDN like Cloudflare. They serve website content to clients using their network of endpoints located all around the world. From the host server perspective though, incoming connections come from the CDN and not directly from the client IP, which can make management a challenge unless the CDN offers tools on their end, like geo restrictions, blacklists, etc.
Elasticsearch
JSON based document-oriented database designed to store, retrieve, and manage document-oriented or semi-structured data. it is schema-less and natively is designed in a distributed fashion, making it extremely easy to expand upon.
Elasticsearch is extremely efficient at handling queries against big data, commonly with a response time of less than a second.
Elasticsearch can also be used as a front end for a traditional relational database using logstash to copy and synchronize records between elasticsearch and the rdbms.
Apache Cassasndra
Leading NoSQL Database solution, offers schema-less non-relational database based on a document rather than a table. Primarily used for big data needs as it was built to scale and offers exceptional response time when compared to a RDMS. NoSQL does have a lack of community support vs mySQL. Cassandra supports a multi-master model, so it is possible to have 100% uptime and it supports CDL, which is similar to SQL query language, but with limitations.
Representational State Transfer (REST API)
REST is a way for two computer systems to communicate over HTTP. Generally the REST API daemon will return a JSON response that the receiving server will parse. REST APIs are generally used programmatically, in other words they are not used directly by people, but by applications and scripts. REST APIs can return just about any data that an application will need, such as user preferences, counts, url strings, etc.
XML vs JSON
JSON
- Javascript object notation
JSON is fairly common today, widely used by many REST API implementations. It is fast, easy to read and parse, and has lots of community support and tools, like Postman. It natively is not secure (but can be used with HTTPS) nor supports any encoding other than UTF-8
XML
Older than JSON, XML is more of a extensible markup language (like HTML) designed to store data. You can define markup elements and create a custom markup language based on XML. XML can support more than just text or numbers (like JSON), it can also support images, charts, graphs, etc. Retrieving data is a bit more difficult than JSON, but XML support multiple types of encodings.
AJAX
- Asynchronous JavaScript And XML
AJAX is a technique for accessing web servers from a web page. Essentially its a way to use XML within Javascript to send/receive data between a client and server. The biggest advantage is the ability to update webpages asynchronously so that parts of a webpage are updated without refreshing the entire page.
http cookie
http cookies are used for a large variety of purposes, but the most common is stateful or arbitrary pieces of information about a user's interaction with a website. Generally you have a separate cookie generated for each website a user accesses, which could even include authentication details, such as a username and whether they are logged in or not.
Cookies have also been used for tracking and dynamic advertising purposes, so a ad generation site, like google ad works, can tailor or customize ads to products you've recently viewed online.
