shell-functools Collection of ideas for new functions

[ ] More string utilities (e.g. regex functions)
[x] Formatting functions (echo 3.1415 | map format "{:.2f}")

Nov 26 '17 19:11 sharkdp

String

[ ] search and match based on regex
[ ] Add regex support for replace input
[x] Add function endwith a pattern

Arithmetic

[ ] Add mod, pow, div and log etc. functions

Filesystem

[ ] Add function mv to move and rename files
[ ] More functions to filter files based on file mode bits and ownship

Nov 28 '17 08:11 leeexyz

@leeleeee Sounds great. Especially the mv function, which will be really useful (we should add cp as well)

Nov 28 '17 19:11 sharkdp

@sharkdp What do you think if we can add du or something like this?

Nov 28 '17 19:11 leeexyz

@sharkdp What do you think if we can add du or something like this?

In which sense? Like a filesize :: Path -> Int function?

By the way, I just realized that the coreutils du commands works very nicely with functools:

> du | filter -c1 greater_than 8000 | map -c2 abspath
8340	/tmp/demo/folder/subdirectory
8408	/tmp/demo/folder
8480	/tmp/demo

Nov 28 '17 20:11 sharkdp

Yeah, more precisely it is filesize :: Path -> String.

$ find . | map basename | filter startswith test_ | map filesize mb 
test_file1 1MB 
test_file2 5.2MB

It is fantastic that linux commands are work very well with these functions. 👍

Nov 28 '17 20:11 leeexyz

filesize is now supported.

Apr 01 '18 20:04 sharkdp

[ ] fst and snd for array-index access (alias for index 1 and index 2)
[ ] replicate
[ ] reverse

Apr 08 '18 18:04 sharkdp

Does split currently work with variable size of whitespace between words? if not, does it make sense to add it? Something like below by default if no separator is provided, then it would be a nice replacement for awk

➜  ~ echo "a b         ccc " | ~/repos/shell-functools/ft/map split | map at 2
ccc

May 05 '18 11:05 ghost

Does split currently work with variable size of whitespace between words?

It does not (yet). We could either provide a split_whitespace function which would simply call .split() without argument and/or provide a split_regex function which could be called with map split_regex '\s+'.

May 06 '18 16:05 sharkdp

The reverse function, do you mean on array level, or on string level?

May 23 '18 08:05 vegarsti

I think array-reverse can be useful. Not sure if we need a string-reverse function.

May 24 '18 07:05 sharkdp

Hey @sharkdp, what about creating a max as well as min function in shell-functools? I can work on something if there is interest.

max :: Array -> Int
find . -name cnt.txt | xargs cat | max

Oct 01 '18 15:10 guilhermeleobas

@guilhermeleobas Sounds great!

Oct 01 '18 19:10 sharkdp

These days, new unix/linux CLI tools are coming out supporting JSON outputs. Would it make sense to create a json function that can parse JSON outputs?

Nov 20 '18 20:11 Jake6329

These days, new unix/linux CLI tools are coming out supporting JSON outputs. Would it make sense to create a json function that can parse JSON outputs?

I've been thinking about JSON support in the past, so this sounds interesting. What would this function do, exactly?

Nov 21 '18 18:11 sharkdp

I can imagine there would a command named json which could be used like this:

$ curl http://echo.jsontest.com/key1/value1 | json key1

The curl command outputs {"key1": "value1"}. The whole command would output value1. The argument to json command would specify the path to the value you want. It could support the full json path spec or just a small part of it.

And then there would also a function called json which could be used like this:

$ curl <url that returns json in each line> | map json <json path>

In this case, the assumption is that the curl outputs a json-encoded string in each line, which is also a proposed standard format called jsonlines. And the map will transform each line with the json function.

$ <command that outputs json> | json <path to an array of objects> | map json <path in each object>

In the above command, the json command narrows data down to an array of objects, and then map json transforms each json into something. I think the path you specify could be some kind of template so that you can transform json into anything, which sounds a bit too ambitious.

Dec 03 '18 20:12 Jake6329

Do you know jq? I think this could probably play well in combination with shell-functools.

Dec 05 '18 08:12 sharkdp

@sharkdp jq seems to cover my use cases. Cool, thanks.

Dec 05 '18 20:12 Jake6329

More functions that should be implemented:

[ ] div/divide
[ ] sum/product (working on arrays)
[ ] matches => similar to starts_width/ends_with

May 18 '19 20:05 sharkdp

Offering

For README.md:

### Usage of `lambda`

The `lambda` *function* allows you to create custom functions using a
definition string to Python's *lambda* function.
``` bash
> seq 3 | map lambda 'x : x+x'
2
4
6

> echo -e 'a:b\n2:4' | map split ':' | map lambda 'x, y : x+y'
ab
6

> seq -3 3 | map lambda 'x : abs(x)'
3
2
1
0
1
2
3

> echo -e "uppercase\nLOWERCASE\ncAMELcASE" | map lambda "x : x.swapcase()"
UPPERCASE
lowercase
CamelCase

For ft/ft/functions.py:

@register("lambda")
@typed(None, None)
def ft_lambda(definition, inp):
    _lambda = 'lambda ' + definition.value
    if type(inp) == list:
        args = list(map(lambda v: v.value, inp))
        return [TypedValue(str((eval(_lambda))(*args)), T_STRING)]
    else:
        return (eval(_lambda))(inp)

May 09 '22 02:05 jdhedden

Offering

For README.md:

### Usage of `shell`

The `shell` *function* allows you to create custom functions using arbitrary
*shell* commands.  If the command string contains pairs curly braces `{}`, the
arguments to the function are formatted into it, otherwise they are appended
to it.
``` bash
> /usr/bin/ls -1 ft/ft/*.py | map shell 'grep -H -c def' | map split ':' | map format '{} has {} functions'
ft/ft/command.py has 14 functions
ft/ft/error.py has 1 functions
ft/ft/functions.py has 60 functions
ft/ft/__init__.py has 0 functions
ft/ft/internal.py has 4 functions
ft/ft/termcolor.py has 2 functions
ft/ft/test_command.py has 5 functions
ft/ft/types.py has 21 functions
ft/ft/version.py has 0 functions

> echo -e 'a\tb\n2\t4' | map shell 'echo "{} and {}"'
a and b
2 and 4

Total lines of text

find | filter is_file | map shell 'cat {} | wc -l' | foldl1 add
5040

For ft/ft/functions.py:

@register("shell")
@typed(None, None)
def shell(command, inp):
    if type(inp) == list:
        args = list(map(lambda v: str(v.value), inp))
    else:
        args = [str(inp)]

    command = command.value
    if '{}' in command:
        command = command.format(*args)
    else:
        command += ' '+' '.join(args)

    result = os.popen(command).read().rstrip('\r\n')

    if type(inp) == list:
        return [TypedValue(result, T_STRING)]
    else:
        return result

May 09 '22 02:05 jdhedden

Offering

For README.md:

### Usage of `regex`

The `regex` *function* permits filtering based on Python supported regular
expressions.  For case-insensitive filtering, use the `(?i:...)` syntax:
``` bash
> /usr/bin/ls | filter regex '(?i:[a-c]+)'
LICENSE
make
path
README.md

For ft/ft/functions.py:

import re

@register("regex")
@typed(T_STRING, T_BOOL)
def contains(pattern, inp):
    pattern = dynamic_cast(T_STRING, pattern).value
    return re.search(pattern, inp)

May 09 '22 02:05 jdhedden

Offering of new command `array` to create or flatten arrays

For README.md:

### Usage of `array`

The `array` command is used to take *lines of input*, and convert them into an
`array`.
``` bash
> seq 3 | array
1	2	3

Given an numerical argument, array will make arrays with that number of columns:

> seq 12 | array 5
1	2	3	4	5
6	7	8	9	10
11	12

Given an argument of '0' (or a non-numerical argument), array will convert arrays back into lines of input:

> echo -e 'aa\tbb\tcc\tdd' | array 0
aa
bb
cc
dd

For ft/array (new):

#!/usr/bin/bash

if [[ -z $1 || $1 -gt 0 ]]; then
    if [[ -n $1 ]]; then
        max=$1
    else
        max=1000000000000000
    fi

    cnt=0
    while read; do
        if [[ $cnt -gt 0 ]]; then
            result+=$'\t'
        fi
        result+=$REPLY

        (( cnt++ ))
        if [[ $cnt -eq $max ]]; then
            echo "$result"
            result=
            cnt=0
        fi
    done

    # Remaining partial line
    if [[ -n $result ]]; then
        echo "$result"
    fi

else
    # array 0
    sed 's@\t@\n@g'
fi

# EOF

May 09 '22 18:05 jdhedden

hi! can u expand on that? like what would these do or what would be their input be like.

[ ] div/divide

[ ] sum/product (working on arrays) - @ sharkdp at https://github.com/sharkdp/shell-functools/issues/7#issuecomment-493706062

Jan 28 '24 14:01 goyalyashpal

Collection of ideas for new functions

Offering

Offering

Offering

Offering of new command array to create or flatten arrays

Offering of new command `array` to create or flatten arrays