Latest revision as of 01:43, 5 June 2024

Common Tools

[edit] Listening Ports

[edit] netstat

~$ netstat -lnp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:17003         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:40333         0.0.0.0:*               LISTEN      -
udp    42240      0 127.0.0.1:8125          0.0.0.0:*                           -
udp    32256      0 169.254.0.2:53          0.0.0.0:*                           -

[edit] Determine process listening port(s) by name

~$ sudo netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      374/apache2

determine the location of the process using the PID

~$ sudo ls -la /proc/374/exe
lrwxrwxrwx 1 root root 0 2020-01-27 00:38 /proc/374/exe -> /usr/lib/apache2/mpm-prefork/apache2

[edit] process activity

[edit] top

~$ top

LoadAvg Load average over 1, 5, and 15 minutes. divide by number of cores to determine load.

[edit] Show snippet in time

~$ top -b -n 1 | grep "load average" -A 15

[edit] ps

ps -AlH

This shows all the processes running, in long format, with the extra full format output

[edit] Network Throughput

[edit] iperf

requires a server/client setup that communicates over port 5001. generally server/client placement is dependent on network accessibility, ie. server would be placed outside of the network, client within. You may need to open 5001 on both sides however, in case established/related is not implicitly accepted from the client side > out.

[edit] Server side

~$ sudo yum install -y iperf
sudo iptables -A INPUT -p tcp --dport 5001 -j ACCEPT -m comment --comment "iperf - 5001"
~$ iperf -s

Example:

$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.1 port 5001 connected with 10.0.0.2 port 47552
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  76.6 MBytes  64.1 Mbits/sec
[  5] local 10.0.0.1 port 5001 connected with 10.0.0.2 port 39914
[  5]  0.0-10.0 sec  84.5 MBytes  70.6 Mbits/sec

[edit] Client side

~$ iperf -c target.ip -p port.number

Example

$ iperf -c 10.0.0.1 -p 5001
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size:  466 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.1 port 39914 connected with 10.0.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  84.5 MBytes  70.8 Mbits/sec

[edit] Disk

[edit] lsblk

Shows a list of block devices. Useful in showing mount points, size, name, etc.

~$ lsblk --all
NAME                        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                         7:0    0        0 loop
loop1                         7:1    0        0 loop
loop2                         7:2    0        0 loop
loop3                         7:3    0        0 loop
loop4                         7:4    0        0 loop
loop5                         7:5    0        0 loop
loop6                         7:6    0        0 loop
loop7                         7:7    0        0 loop
sr0                          11:0    1 1024M  0 rom
sda                           8:0    0    8G  0 disk
├─sda1                        8:1    0  500M  0 part /boot
└─sda2                        8:2    0  7.5G  0 part
  ├─VolGroup-lv_root (dm-0) 253:0    0  6.7G  0 lvm  /
  └─VolGroup-lv_swap (dm-1) 253:1    0  816M  0 lvm  [SWAP]

[edit] fdisk

Tool for showing partitions and disks, as well as other things

~$ sudo fdisk -l

Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000091fd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64        1045     7875584   8e  Linux LVM

Disk /dev/mapper/VolGroup-lv_root: 7205 MB, 7205814272 bytes
255 heads, 63 sectors/track, 876 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/VolGroup-lv_swap: 855 MB, 855638016 bytes
255 heads, 63 sectors/track, 104 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

[edit] df

shows the amount of disk space on the file system and the file system type. Can also show inodes.

~$ df -Th
Filesystem              Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root xfs       6.2G  1.5G  4.8G  24% /
devtmpfs                devtmpfs  485M     0  485M   0% /dev
tmpfs                   tmpfs     496M     0  496M   0% /dev/shm
tmpfs                   tmpfs     496M  6.8M  490M   2% /run
tmpfs                   tmpfs     496M     0  496M   0% /sys/fs/cgroup
/dev/sda1               xfs      1014M  159M  856M  16% /boot
tmpfs                   tmpfs     100M     0  100M   0% /run/user/1000

[edit] du

shows the amount of space used from a specific directory and recursively includes its children.

~$ du -h /home/user/temp
4.0K	/home/user/temp

Find the largest 20 files/folders on a system

$ sudo du -a / 2>/dev/null | sort -n -r | head -n 20
1893852	/
990024	/var
447452	/home
447448	/home/r00t
429228	/home/r00t/backup
429224	/home/r00t/backup/4_2_2019
423912	/usr
421432	/var/www
421420	/var/www/mediawiki-1.19.2
362128	/home/r00t/backup/4_2_2019/wikidb4_2_2019.tgz
277372	/var/www/mediawiki-1.19.2/images
273504	/var/www/mediawiki-1.19.2/images/upload
251020	/var/lib
223472	/var/cache
219792	/var/cache/apt
202204	/usr/share
188916	/var/cache/apt/archives
171208	/var/lib/mysql
157700	/var/lib/mysql/ibdata1
128896	/var/www/mediawiki-1.19.2/images/upload/20140222_162517.mp4

Find the largest files on the system

~$ sudo find / -type f -exec du -b {} + 2>/dev/null | sort -rn | head -n 20
222067408	/usr/lib/chromium-browser/chromium-browser
204799654	/root/.pm2/logs/mm-error.log
179328456	/root/.pm2/pm2.log
122519636	/opt/MagicMirror/node_modules/electron/dist/electron
110013816	/var/cache/apt/archives/chromium-browser_120.0.6099.102-rpt1_armhf.deb
104857600	/var/swap
91840204	/root/.pm2/logs/mm-out.log
91809683	/home/pi/.cache/electron/d571833ca78195032b264d2a913d36a9a912eb0d7f2ae3c7b66b4c026b42e5ac/electron-v29.1.6-linux-armv7l.zip
89928228	/root/.cache/electron/286d40817318e11a97a0cee04bdcf6048fe8051ec36b5aad5b09e3be0c5d2eae/electron-v27.2.0-linux-armv7l.zip
80846180	/usr/bin/node
74259420	/usr/lib/arm-linux-gnueabihf/libLLVM-11.so.1
68711385	/var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_bullseye_main_binary-armhf_Packages
67108864	/var/log/journal/92a1797f796b4d8783385896e21869d8/[email protected]~
56714348	/usr/lib/arm-linux-gnueabihf/libwebkit2gtk-4.0.so.37.67.5
50331648	/var/log/journal/92a1797f796b4d8783385896e21869d8/[email protected]~
45251556	/usr/lib/pypy/bin/libpypy-c.so
40381082	/usr/lib/chromium-browser/local_rustc_sysroot/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcore.rlib
35954684	/home/pi/Bookshelf/BeginnersGuide-4thEd-Eng_v2.pdf
33785834	/var/cache/apt/pkgcache.bin
33723805	/var/cache/apt/srcpkgcache.bin

[edit] local mounts and file systems

~$ sudo cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,seclabel,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,relatime,size=499208k,nr_inodes=124802,mode=755 0 0
devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,seclabel,relatime 0 0
/dev/mapper/VolGroup-lv_root / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /selinux selinuxfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,relatime,size=499208k,nr_inodes=124802,mode=755 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

[edit] Testing

Storage Device Burn-In
Spinning disk hard drives have moving parts, by definition. These parts are highly-sensitive to shock and vibration, and will eventually wear out with use. Consider pre-flighting every storage device before putting it into production, paying attention to:

Start a long HDD self test (smartctl -t long /dev/)
After the test is done (could take 12+ hours), check the results (smartctl -a /dev/)
Pending sector reallocations (smartctl -a /dev/ | grep Current_Pending_Sector)
Reallocated sector count (smartctl -a /dev/ | grep Reallocated_Sector_Ct)
UDMA CRC errors (smartctl -a /dev/ | grep UDMA_CRC_Error_Count)
HDD and SSD write latency consistency (diskinfo -wS ) Unformatted drives only!
HDD and SSD hours (smartctl -a /dev/ | grep Power_On_Hours)
NVMe percentage used (nvmecontrol logpage -p 2 nvme0 | grep “Percentage used”)
Take time before deploying the system to create a pool and subject it to as close to the real-world workload as possible. This can reveal individual drive issues but also can help determine if an alternative pool layout is better suited to that workload. Be cautious of used drives as vendors may not be honest or informed about the age and health of any given drive. Check the number of hours on all “new” drives using smartctl(8) as they may in fact be “recertified” or simply untruthfully advertised. A drive vendor may also zero the hours of a drive during recertification, masking its true age. All storage devices sold by iXsystems are tested for a minimum of 48 hours prior to shipment.

[edit] hardware/driver

[edit] dmesg

Used to gather information about hardware and drivers on a system but can also be used to troubleshoot and show verbose logs pertaining to such.

$ dmesg | less
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.11.0-13-generic (buildd@aatxe) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #20-Ubuntu SMP Wed Oct 23 17:26:33 UTC 2013 
(Ubuntu 3.11.0-13.20-generic 3.11.6)
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   NSC Geode by NSC
[    0.000000]   Cyrix CyrixInstead
[    0.000000]   Centaur CentaurHauls
[    0.000000]   Transmeta GenuineTMx86
[    0.000000]   Transmeta TransmetaCPU
[    0.000000]   UMC UMC UMC UMC
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007dc08bff] usable

Find specific device

~$ dmesg | grep sda

[    1.280971] sd 2:0:0:0: [sda] 488281250 512-byte logical blocks: (250 GB/232 GiB)
[    1.281014] sd 2:0:0:0: [sda] Write Protect is off
[    1.281016] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.281039] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.359585]  sda: sda1 sda2 < sda5 sda6 sda7 sda8 >
[    1.360052] sd 2:0:0:0: [sda] Attached SCSI disk

[edit] Downloading

[edit] wget

[edit] quiet

-q | --quiet

Turn off Wget's output

[edit] verbose

-v | --verbose

Turn on verbose output

[edit] input file

-i | --input-file=file

Reads URLs from a local or external file.

[edit] Tries

-t | --tries=number

number of times to retry.

[edit] Output

-O | --output-document=file

all documents will be concatenated together and written to the file

[edit] Progress Indicator

--progress=type

Show a process indicator with varying types, such as dot and bar

[edit] CLI

[edit] telnet

[edit] example: HTTP request

Below is a manual telnet example:

~$ telnet stackoverflow.com 80
Trying 151.101.65.69...
Connected to stackoverflow.com.
Escape character is '^]'.
GET /questions HTTP/1.0
Host: stackoverflow.com

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
...

[edit] find

Find the largest files in a directory

$ find $HOME -type f -printf '%s %p\n' | sort -nr | head -10
370813860 /home/r00t/backup/4_2_2019/wikidb4_2_2019.tgz
68699129 /home/r00t/backup/4_2_2019/wikidb4_2_2019.sql
18266096 /home/r00t/downloads/mediawiki-1.19.2.tar.gz
55320 /home/r00t/downloads/Vector-MW1.19-112648.tar.gz
40273 /home/r00t/.bash_history
26023 /home/r00t/temp/website/assistfuncs.js
22139 /home/r00t/Banned_IPs.txt
14919 /home/r00t/temp/website/image_db.html
11409 /home/r00t/.viminfo
10273 /home/r00t/temp/website/softwareupdate.html

[edit] ulimit

determine max number of processes per user.

~$ ulimit -a | grep processes
max user processes              (-u) unlimited

[edit] zip

compress multiple folders (or one)

~$ zip -r <output_file> <folder_1> <folder_2> ... <folder_n>

[edit] temp

[edit] http get

parameters are passed inline with the URL request.

http://test.com?param=foo,param=bar

[edit] http post

parameters are passed within the body

~$ telnet test.com 80
POST /index.html HTTP/1.1
Host: test.com
param=foo, param=bar

[edit] javascript sop

Same-Origin Policy (SOP)

common when using javascript to integrated with external resources, like an API.
there are security measures within browsers that restrict interaction between documents (or scripts) that have different origins.

[edit] decrypt ssl

[edit] client

use pre-master key captured by the client side using an environment variable populated when a browser visits a SSL site. Afterwards used in wireshark to decrypt SSL traffic sent to/from the client.

[edit] server

from the server side it can be a bit more complicated depending on the type of encryption.
SSL decryption works only if the key exchange was of type RSA or static DH; with "DHE" and "ECDHE" cipher suites, you won't be able to decrypt such a session, even with knowledge of the server private key. In that case, you will need either the negotiated "master secret", or to use the server private key to actively intercept the connection (in a Man-in-the-Middle setup).

[edit] Akamai (CDN)

Akamai is a CDN like Cloudflare. They serve website content to clients using their network of endpoints located all around the world. From the host server perspective though, incoming connections come from the CDN and not directly from the client IP, which can make management a challenge unless the CDN offers tools on their end, like geo restrictions, blacklists, etc.

[edit] Elasticsearch

JSON based document-oriented database designed to store, retrieve, and manage document-oriented or semi-structured data. it is schema-less and natively is designed in a distributed fashion, making it extremely easy to expand upon.
Elasticsearch is extremely efficient at handling queries against big data, commonly with a response time of less than a second.
Elasticsearch can also be used as a front end for a traditional relational database using logstash to copy and synchronize records between elasticsearch and the rdbms.

[edit] Apache Cassasndra

Leading NoSQL Database solution, offers schema-less non-relational database based on a document rather than a table. Primarily used for big data needs as it was built to scale and offers exceptional response time when compared to a RDMS. NoSQL does have a lack of community support vs mySQL. Cassandra supports a multi-master model, so it is possible to have 100% uptime and it supports CDL, which is similar to SQL query language, but with limitations.

[edit] Representational State Transfer (REST API)

REST is a way for two computer systems to communicate over HTTP. Generally the REST API daemon will return a JSON response that the receiving server will parse. REST APIs are generally used programmatically, in other words they are not used directly by people, but by applications and scripts. REST APIs can return just about any data that an application will need, such as user preferences, counts, url strings, etc.

[edit] XML vs JSON

[edit] JSON

Javascript object notation

JSON is fairly common today, widely used by many REST API implementations. It is fast, easy to read and parse, and has lots of community support and tools, like Postman. It natively is not secure (but can be used with HTTPS) nor supports any encoding other than UTF-8

[edit] XML

Older than JSON, XML is more of a extensible markup language (like HTML) designed to store data. You can define markup elements and create a custom markup language based on XML. XML can support more than just text or numbers (like JSON), it can also support images, charts, graphs, etc. Retrieving data is a bit more difficult than JSON, but XML support multiple types of encodings.

[edit] AJAX

Asynchronous JavaScript And XML

AJAX is a technique for accessing web servers from a web page. Essentially its a way to use XML within Javascript to send/receive data between a client and server. The biggest advantage is the ability to update webpages asynchronously so that parts of a webpage are updated without refreshing the entire page.

[edit] http cookie

http cookies are used for a large variety of purposes, but the most common is stateful or arbitrary pieces of information about a user's interaction with a website. Generally you have a separate cookie generated for each website a user accesses, which could even include authentication details, such as a username and whether they are logged in or not.
Cookies have also been used for tracking and dynamic advertising purposes, so a ad generation site, like google ad works, can tailor or customize ads to products you've recently viewed online.