Using grep as highlighter

Posted on December 18, 2015 by ronaldpringadi
1
$ grep --color -E '^|pattern1|pattern2' file name
Posted in Bash, Linux | Comments Off on Using grep as highlighter

TCL programming

Posted on November 12, 2015 by ronaldpringadi

A reusable expect dispatcher I kept around for running the same kind of operation across a list of servers — untar an index, restart a service, patch a config file. The trick is that the script reads the first command-line argument and dispatches to a same-named proc, so one script holds many small ops:

1
2
3
./script.exp UntarLuceneindexes
./script.exp stopLuceneServices
./script.exp startLuceneServices

That’s the $functionName line at the bottom — it literally calls whatever proc name you passed in. Crude, but very handy when you’re iterating on a runbook.

The script below assumes you’ve set up SSH keys so ssh root@host connects without a password prompt. That keeps secrets out of the script and the shell history.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
#!/usr/bin/expect -f
# Expect dispatcher. Beware: whitespace matters in Tcl.

# Multi-line comments use a set / curly-brace trick:
# set comment {
#     your multi-line comment
# }

# ---------------------------------------------------
# Function definitions
# ---------------------------------------------------

if {[llength $argv] == 0} {
    send_user "Usage: script.exp FUNCTION_NAME\n"
    send_user "Example: script.exp UntarLuceneindexes\n"
    exit 1
}
set functionName [lindex $argv 0]

# Servers used by the Lucene-related procs.
set aLuceneServers {your_server_1 your_server_2 your_server_3}

proc printPass {} {
    send_user "\n\[PASS\]\n"
}

proc printFail {} {
    send_user "\n\[FAIL\]\n"
    exit 1
}

proc simpleTest {} {
    set aProductionServers {your_server_1 your_server_2 your_server_3 your_server_4 your_server_5 your_server_6}
    foreach host $aProductionServers {
        send_user "Processing ... '$host'\n"
        set timeout 60
        spawn ssh root@$host

        send "hostname\r"
        expect $host

        send "uptime\r"
        expect "load average"

        send "exit\r"
    }
}

proc UntarLuceneindexes {} {
    global aLuceneServers
    set sCurrentLuceneTar "index2_20151009.tar.bz2"
    foreach host $aLuceneServers {
        send_user "\n================================================\n"
        send_user "Processing ... '$host'\n"
        send_user "================================================\n"
        set timeout 60
        spawn ssh root@$host

        send "hostname\r"
        expect $host

        send "su - path\r"
        send "pwd\r"
        expect "/home/path"

        send "ls $sCurrentLuceneTar | wc -l\r"
        expect "1"

        send "tar -xjvf $sCurrentLuceneTar &\r"
        send "sleep 2\r"
        send "ps aux | grep index2 | grep tar | wc -l\r"
        send "disown %1\r"
        expect "1"

        send_user "Done processing ... '$host'\n"
    }
}

proc updateDBRef {} {
    set apathServers {your_server_6}
    set newDB "some_db_20150925"
    foreach host $apathServers {
        send_user "\n================================================\n"
        send_user "Processing ... '$host'\n"
        send_user "================================================\n"
        set timeout 60
        spawn ssh root@$host

        send "hostname\r"
        expect $host

        send "su - path\r"
        send "pwd\r"
        expect "/home/path"

        set propsFile "/home/path/path-current/SomeConfig.properties"
        send "grep -v 'com.somecompany.setting.server.db.pathdb2.url=' $propsFile > $propsFile.new\r"
        send "echo 'com.somecompany.setting.server.db.pathdb2.url=jdbc:postgresql://127.0.0.1:5432/$newDB' >> $propsFile.new\r"
        send "mv $propsFile.new $propsFile\r"

        send_user "Done processing ... '$host'\n"
    }
}

proc stopLuceneServices {} {
    global aLuceneServers
    foreach host $aLuceneServers {
        send_user "\n================================================\n"
        send_user "Processing ... '$host'\n"
        send_user "================================================\n"
        set timeout 60
        spawn ssh root@$host

        send "hostname\r"
        expect $host

        send "service lucene stop\r"
        expect {
            "stopped PID"           { printPass }
            "lucene is not running" { printPass }
            timeout                 { printFail }
        }

        send_user "Done processing ... '$host'\n"
    }
}

proc startLuceneServices {} {
    global aLuceneServers
    foreach host $aLuceneServers {
        send_user "\n================================================\n"
        send_user "Processing ... '$host'\n"
        send_user "================================================\n"
        set timeout 60
        spawn ssh root@$host

        send "hostname\r"
        expect $host

        send "service lucene start\r"
        expect {
            "started PID" { printPass }
            timeout       { printFail }
        }

        send_user "Done processing ... '$host'\n"
    }
}

# ---------------------------------------------------
# Run
# ---------------------------------------------------

set timeout 60
log_file -noappend expect.log  ;# default is append; -noappend overwrites

$functionName

Tcl or Expect? The post title says Tcl, but almost every interesting line above — spawn, expect, send, send_user, log_file, the #!/usr/bin/expect -f shebang — comes from Expect, an extension that sits on top of Tcl and lets you drive interactive programs (SSH, telnet, vendor CLIs) by pattern-matching on their output. Pure Tcl is the language; Expect is what makes a script like this one possible. 🤖

Use SSH keys, not expect-and-send-password. The original version of this script had blocks like expect “*assword: “ followed by send “YOUR_SERVER_PASSWORD\r”. That works, but it puts a plaintext password in your script and your shell history. The cleaner answer is SSH keypair auth — the script above assumes that’s already set up, so the password block disappears entirely. Reserve the expect-and-send-password pattern for the cases where it’s truly unavoidable: old network gear, vendor CLIs, devices that don’t accept keys.

You don’t always need Expect. If all you need is to run the same shell command across a list of servers, plain SSH in a loop is simpler and easier to debug:

1
2
3
for host in your_server_1 your_server_2 your_server_3; do
  ssh "$host" "service lucene stop"
done

For larger fleets, parallel-ssh (pssh) or ansible -m shell -a “…” mygroup are usually saner. Expect earns its keep when you have to navigate an interactive prompt that doesn’t accept piped commands — confirm dialogs, paged output, vendor consoles. 💡

Posted in Linux, TCL/Expect | Comments Off on TCL programming

Simple unit test is bash file

Posted on October 28, 2015 by ronaldpringadi

Consider the following 3 files:

1. shellTestFramework.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#!/bin/bash
# Copyright (c) Ronald Pringadi

# Before each Test
function setUpTest(){
    #"Please overwrite this function on your unit test. Something that need to be done before each test"
    return 0;
}

function tearDownTest(){
    #"Please overwrite this function on your unit test. Something that need to be done after each test"
    return 0;
}


# Helping assertion
function assertEqual(){
    if [ "$1" == "$2" ]; then
        echo "`caller 0`. PASS"
    else
        echo "`caller 0`. FAIL. $1!=$2"
    fi
}

# Helping assertion
function assertNotEqual(){
    if [ "$1" != "$2" ]; then
        echo "`caller 0`. PASS"
    else
        echo "`caller 0`. FAIL. $1==$2"
    fi
}

# To run all test.
# Will run 'setUpTest()' before each test.
# Will run 'tearDownTest()' after each test.
function runAllTest(){
    array=($(compgen -A function |grep -i test))
    for i in "${array[@]}"
    do
       if [ "$i" != "setUpTest" ] && [ "$i" != "runAllTest" ] && [ "$i" != "tearDownTest" ]; then
        echo "`setUpTest`"
        echo "`$i`"
        echo "`tearDownTest`"
       fi
    done
}

2. osdetector.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
#!/bin/bash
# Copyright (c) Ronald Pringadi

# Returns String ("ubuntu", "censtos", "redhat", "unknown")
function detectAnyOS(){
    OS="unknown"
   
    DETECT="ubuntu"
    if [ `detectSpecificOS "$DETECT"` -eq 1 ] ; then
        echo $DETECT
        return 0
    fi
   
    DETECT="centos"
    if [ `detectSpecificOS "$DETECT"` -eq 1 ] ; then
        echo $DETECT
        return 0
    fi
   
    DETECT="Red Hat"
    if [ `detectSpecificOS "$DETECT"` -eq 1 ] ; then
        echo "redhat"
        return 0
    fi
   
    DETECT="RHEL"
    if [ `detectSpecificOS "$DETECT"` -eq 1 ] ; then
        echo "redhat"
        return 0
    fi
}

# Input String ("ubuntu", "Centos", "RedHat")
# Returns String "1"=found, "0"=not found
function detectSpecificOS(){
    DETECT=$1
    IS_FOUND=`cat /etc/*-release |grep -i "$DETECT"|wc -l`
    if [ $IS_FOUND -gt 0 ]; then
        echo "1"
    else
        echo "0"
    fi
}

# Returns String ("6", "5.5", "4")
function detectOSVersion(){
    # Works for ubuntu
    VERSION=`cat /etc/*-release |grep 'DISTRIB_RELEASE='|grep -o '[0-9|.]\+'|head -1`
    if [ "$VERSION" != "" ]; then
        echo $VERSION
        return 0
    fi
   
    # Works for Centos
    VERSION=`cat /etc/*-release |grep ' release '|grep -o '[0-9|.]\+'|head -1`
    if [ "$VERSION" != "" ]; then
        echo $VERSION
        return 0
    fi
   
    # Final resort
    VERSION=`cat /etc/*-release |grep -i 'ubuntu\|red hat\|centos'| grep -o '\ [0-9|.]\+'|head -1`
    if [ "$VERSION" != "" ]; then
        echo $VERSION
        return 0
    fi
   
}


# Returns String ("6", "5", "4")
function detectOSMajorVersion(){
    VERSION=`detectOSVersion`
    MAJOR_VERSION=`echo "$VERSION" |grep -o '[0-9]\+'|head -1`
    echo "$MAJOR_VERSION"
}

# Input String OS  ("ubuntu", "Centos", "RedHat")
# Input String OSVersion name
# Returns String "1"=supported, "0"=not supported
function isOSSupported(){
    SUPPORTED="0"
    OS=`echo "$1"| tr '[:upper:]' '[:lower:]'`
    VERSION="$2"
    MAJOR_VERSION=`echo "$VERSION" |grep -o '[0-9]\+'|head -1`
    if [ "$OS" == "ubuntu" ]; then
        if [ "$MAJOR_VERSION" == "12" ] || [ "$MAJOR_VERSION" == "13" ] || [ "$MAJOR_VERSION" == "14" ]; then
            SUPPORTED="1"
        fi
    fi
    if [ "$OS" == "red hat" ] || [ "$OS" == "redhat" ] || [ "$OS" == "centos" ]; then
        if [ "$MAJOR_VERSION" == "6" ]; then
            SUPPORTED="1"
        fi
    fi
    echo $SUPPORTED
}

3. osdetector.test.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
#!/bin/bash
# Copyright (c) Ronald Pringadi
my_dir="$(dirname "$0")"
source "$my_dir/osdetector.sh"
source "$my_dir/shellTestFramework.sh"


# Before each Test
function setupTest(){
    RELEASE_CONTENT="Ubuntu 14.04.3 LTS"
    echo "$RELEASE_CONTENT">/etc/test-release
}

function adetectSpecificOSTest(){
    DETECT="Ubuntu"
    RESULT=`detectSpecificOS $DETECT`
    assertEqual "$RESULT" "1"
   
    DETECT="Centos"
    RESULT=`detectSpecificOS $DETECT`
    assertEqual "$RESULT" "0"
}


function detectAnyOSTest(){
    RESULT=`detectAnyOS`
    assertEqual "$RESULT" "ubuntu"
    assertNotEqual "$RESULT" "centos"
}

function detectOSAndOsVersionTest(){   
    RELEASE_CONTENT="Ubuntu 14.04.3 LTS"
    echo "$RELEASE_CONTENT" > /etc/test-release
    RESULT=`detectOSVersion`
    assertEqual "$RESULT" "14.04.3"
    RESULT=`detectOSMajorVersion`
    assertEqual "$RESULT" "14"
    RESULT=`detectAnyOS`
    assertEqual "$RESULT" "ubuntu"
   
    RELEASE_CONTENT="CentOS release 6.6 (Final)"
    echo "$RELEASE_CONTENT" > /etc/test-release
    RESULT=`detectOSVersion`
    assertEqual "$RESULT" "6.6"
    RESULT=`detectOSMajorVersion`
    assertEqual "$RESULT" "6"
    RESULT=`detectAnyOS`
    assertEqual "$RESULT" "centos"
   
    RELEASE_CONTENT="CentOS Linux release 7.1.1503 (Core)"
    echo "$RELEASE_CONTENT" > /etc/test-release
    RESULT=`detectOSVersion`
    assertEqual "$RESULT" "7.1.1503"
    RESULT=`detectOSMajorVersion`
    assertEqual "$RESULT" "7"
    RESULT=`detectAnyOS`
    assertEqual "$RESULT" "centos"
   
    RELEASE_CONTENT="Red Hat Enterprise Linux Server release 6.3 (Santiago)"
    echo "$RELEASE_CONTENT" > /etc/test-release
    echo "$RELEASE_CONTENT" >> /etc/test-release
    RESULT=`detectOSVersion`
    assertEqual "$RESULT" "6.3"
    RESULT=`detectOSMajorVersion`
    assertEqual "$RESULT" "6"
    RESULT=`detectAnyOS`
    assertEqual "$RESULT" "redhat"
}

function isOSSupportedTest(){  
   
    RESULT=`isOSSupported "ubuntu" "11"`
    assertEqual "$RESULT" "0"
   
    RESULT=`isOSSupported "ubuntu" "12"`
    assertEqual "$RESULT" "1"
   
    RESULT=`isOSSupported "ubuntu" "13"`
    assertEqual "$RESULT" "1"
   
    RESULT=`isOSSupported "ubuntu" "14"`
    assertEqual "$RESULT" "1"
   
    RESULT=`isOSSupported "ubuntu" "15"`
    assertEqual "$RESULT" "0"
       
    RESULT=`isOSSupported "redhat" "5"`
    assertEqual "$RESULT" "0"
   
    RESULT=`isOSSupported "redhat" "6"`
    assertEqual "$RESULT" "1"
   
    RESULT=`isOSSupported "redhat" "7"`
    assertEqual "$RESULT" "0"
   
    RESULT=`isOSSupported "red hat" "5"`
    assertEqual "$RESULT" "0"
   
    RESULT=`isOSSupported "red hat" "6"`
    assertEqual "$RESULT" "1"
   
    RESULT=`isOSSupported "red hat" "7"`
    assertEqual "$RESULT" "0"
       
    RESULT=`isOSSupported "centos" "5"`
    assertEqual "$RESULT" "0"
   
    RESULT=`isOSSupported "centos" "6"`
    assertEqual "$RESULT" "1"
   
    RESULT=`isOSSupported "centos" "7"`
    assertEqual "$RESULT" "0"
   
    RESULT=`isOSSupported "Fedora" "50"`
    assertEqual "$RESULT" "0"
   
}


runAllTest
Posted in Bash | Comments Off on Simple unit test is bash file

Multithreading in Java using ThreadPoolExecutor

Posted on September 23, 2015 by ronaldpringadi

ThreadWorker is your custom class.

1
2
3
4
5
6
7
8
9
try {
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(MAX_THREAD_SIZE);
for (int i = 1; i Random randomGenerator = new Random();
executor.submit(new ThreadWorker("worker" + i, randomGenerator.nextInt(10)));
LOG.info(i);
}
} catch (Exception e) {
LOG.error("Hmm something is not right.", e);
}
Posted in java | Comments Off on Multithreading in Java using ThreadPoolExecutor

Getting the caller method details using Java

Posted on September 23, 2015 by ronaldpringadi
1
2
3
4
5
6
7
8
9
10
 public static String getCallerClassName() {
        StackTraceElement[] stElements = Thread.currentThread().getStackTrace();
        for (int i=1; i<stElements.length; i++) {
            StackTraceElement ste = stElements[i];
            if (!ste.getClassName().equals(YOUR_CURRENTCLASS.class.getName()) && ste.getClassName().indexOf("java.lang.Thread")!=0) {
                return ste.getClassName()+"."+ ste.getMethodName()+"()";
            }
        }
        return null;
     }
Posted in java | Comments Off on Getting the caller method details using Java

Compare File Permission Recussively on Linux Directories

Posted on August 13, 2015 by ronaldpringadi

Scan

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/perl

use File::Find;

my $directory1 = '/root/rpmbuild/RPMSX';
my $directory2 = '/root/rpmbuild/RPMSX.bak';

find(\&hashfiles, $directory1);

sub hashfiles {
  my $file1 = $File::Find::name;
  (my $file2 = $file1) =~ s/^$directory1/$directory2/;
        return(0) if (! -f $file2) ;
  my $mode1 = (stat($file1))[2] ;
  my $mode2 = (stat($file2))[2] ;

  my $uid1 = (stat($file1))[4] ;
  my $uid2 = (stat($file2))[4] ;

  print "Permissions for $file1 and $file2 are not the same\n" if ( $mode1 != $mode2 );
  print "Ownership for $file1 and $file2 are not the same\n" if ( $uid1 != $uid2 );
}
Posted in Linux | Comments Off on Compare File Permission Recussively on Linux Directories

Linux find files and total their size

Posted on July 22, 2015 by ronaldpringadi
1
find ./ -type f -newerct "1 May 2015" ! -newerct "1 Jul 2015" -print0 | du --files0-from=- -hc| tail -n1
Posted in Bash, Linux | Comments Off on Linux find files and total their size

Wireshark filters

Posted on June 22, 2015 by ronaldpringadi

Filter by ip dst or source using wildcard on the last 3 digits:

(ip.dst == 192.168.0.0/24) || (ip.src == 192.168.0.0/24)

Posted in Linux | Comments Off on Wireshark filters

MSBuild common errors and how to fix them

Posted on April 16, 2015 by ronaldpringadi

Building with MSBuild produced an error or warning

Run MSBuild with (d)etailed verbose mode and capture the output to a file:

1
msbuild someproject.csproj /t:Clean;Build;Transfer /p:OutputPath=bin\autobuild;BuildNumber=-1;Configuration=Debug /v:d >build.txt

Open build.txt and search for the MSBxxxx error or warning code. For example:

1
2
Consider app.config remapping of assembly "DotNetOpenAuth.AspNet, Culture=neutral, PublicKeyToken=2780ccd10d57b246" from Version "4.0.0.0" [] to Version "4.1.0.0" [C:\somedir\bin\DotNetOpenAuth.AspNet.dll] to solve conflict and get rid of warning.
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets(1605,5): warning MSB3247: Found conflicts between different versions of the same dependent assembly.

If it doesn’t already exist, add an App.config file with the following content:

1
2
3
4
5
6
7
8
9
10
11
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <runtime>
        <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
            <dependentAssembly>
                <assemblyIdentity name="DotNetOpenAuth.AspNet" publicKeyToken="2780ccd10d57b246" />
                <bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.1.0.0" />
            </dependentAssembly>
        </assemblyBinding>
    </runtime>
</configuration>

Also change your .csproj to include the App.config file:

1
2
3
<ItemGroup>
    <None Include="App.config" />
</ItemGroup>

A few useful additions.

This fix applies to classic .NET Framework, not .NET Core / .NET 5+. Binding redirects in App.config are a .NET Framework concept. Modern .NET (Core 3.x, 5, 6, 7, 8, 9, …) resolves assembly versions through *.deps.json and unifies them automatically — you don’t write binding redirects, and MSB3247 generally doesn’t show up the same way. If your project is <Project Sdk=”Microsoft.NET.Sdk”> rather than the legacy XML format, you’re on modern .NET and this post doesn’t apply. 🧭

A better way to inspect a build than reading build.txt. Pass /bl (or -bl) to MSBuild instead of (or in addition to) /v:d:

1
msbuild someproject.csproj /t:Build /bl

This produces an msbuild.binlog file with the full structured build log. Open it in the free MSBuild Structured Log Viewer and you get a navigable tree of every target, task, and property, with full search — a vastly better experience than grepping through a text file. /v:diag is the next step up in plain-text verbosity if you don’t want a binlog (and is what the docs call “diagnostic” level).

Two other MSBuild errors that come up a lot.

MSB3644 — “Reference assemblies for .NETFramework,Version=vX.X were not found”. Classic on a fresh build agent. The targeting pack for the .NET Framework version your project asks for isn’t installed. Two fixes: install the matching .NET Framework Targeting Pack via the Visual Studio Installer (Individual components → “.NET Framework X.X targeting pack”), or downgrade <TargetFrameworkVersion> in your .csproj to a version that is installed.

MSB4018 — “The X task failed unexpectedly”. The MSBuild equivalent of “something blew up, here’s the stack trace”. Usually one of: a custom task that crashed, an SDK version mismatch (your global.json pins an SDK that isn’t on the machine), or a corrupted obj/ folder from an interrupted build. The standard checklist: delete bin/ and obj/, run dotnet restore (or nuget restore for legacy), confirm dotnet –version matches what global.json demands, then rebuild. If it still fails, the binlog from the previous tip will show you which task threw — that’s nearly always the real clue. 🛠️

Posted in C# | Tagged | Comments Off on MSBuild common errors and how to fix them

Allowing a linux/unix user all sudo access without password

Posted on December 15, 2014 by ronaldpringadi

/etc/sudoers

yourusername ALL=(ALL) NOPASSWD:ALL

Posted in Linux | Tagged , | Comments Off on Allowing a linux/unix user all sudo access without password