Advanced features

Here are yet more tools that the average user won’t need, but might come in handy one day.

Click here to open an interactive version of this notebook.

Nested dictionaries

Nested dictionaries are a useful way of storing complex data (and in fact are more or less the basis of JSON), but can be a pain to interact with if you don’t know the structure in advance. Sciris has several functions for working with nested dictionaries. For example:

import sciris as sc

# Create the structure
nest = {}
sc.makenested(nest, ['key1','key1.1'])
sc.makenested(nest, ['key1','key1.2'])
sc.makenested(nest, ['key1','key1.3'])
sc.makenested(nest, ['key2','key2.1','key2.1.1'])
sc.makenested(nest, ['key2','key2.2','key2.2.1'])

# Set the value for each "twig"
count = 0
for twig in sc.iternested(nest):
    count += 1
    sc.setnested(nest, twig, count)

# Convert to a JSON to view the structure more clearly
sc.printjson(nest)

# Get all the values from the dict
values = []
for twig in sc.iternested(nest):
    values.append(sc.getnested(nest, twig))
print(f'{values = }')

{
  "key1": {
    "key1.1": 1,
    "key1.2": 2,
    "key1.3": 3
  },
  "key2": {
    "key2.1": {
      "key2.1.1": 4
    },
    "key2.2": {
      "key2.2.1": 5
    }
  }
}
values = [1, 2, 3, 4, 5]

Sciris also has a sc.search() function, which can find either keys or values that match a certain pattern:

print(sc.search(nest, 'key2.1.1'))
print(sc.search(nest, value=5))

[['key2', 'key2.1', 'key2.1.1']]
[['key2', 'key2.2', 'key2.2.1']]

There’s even an sc.iterobj() function that can make arbitrary changes to an object:

def increment(obj):
    return obj + 1000 if isinstance(obj, int) and obj !=3 else obj

sc.iterobj(nest, increment, inplace=True)
sc.printjson(nest)

{
  "key1": {
    "key1.1": 1001,
    "key1.2": 1002,
    "key1.3": 3
  },
  "key2": {
    "key2.1": {
      "key2.1.1": 1004
    },
    "key2.2": {
      "key2.2.1": 1005
    }
  }
}

Context blocks

Sciris contains two context block (i.e. “with ... as”) classes for catching what happens inside them.

sc.capture() captures all text output to a variable:

import sciris as sc
import numpy as np

def verbose_func(n=200):
    for i in range(n):
        print(f'Here are 5 random numbers: {np.random.rand(5)}')

with sc.capture() as text:
    verbose_func()

lines = text.splitlines()
target = '777'
for l,line in enumerate(lines):
    if target in line:
        print(f'Found target {target} on line {l}: {line}')

Found target 777 on line 35: Here are 5 random numbers: [0.30647147 0.51483255 0.77745685 0.30184163 0.30430842]
Found target 777 on line 88: Here are 5 random numbers: [0.70554215 0.47936335 0.05371795 0.39730575 0.88777415]
Found target 777 on line 105: Here are 5 random numbers: [0.53431613 0.32916132 0.89777977 0.43933727 0.54951866]
Found target 777 on line 116: Here are 5 random numbers: [0.05608838 0.66205077 0.49744291 0.97772645 0.68897064]

The other function, sc.tryexcept(), is a more compact way of writing try ... except blocks, and gives detailed control of error handling:

def fickle_func(n=1):
    for i in range(n):
        rnd = np.random.rand()
        if rnd < 0.005:
            raise ValueError(f'Value {rnd:n} too small')
        elif rnd > 0.99:
            raise RuntimeError(f'Value {rnd:n} too big')

sc.heading('Simple usage, exit gracefully at first exception')
with sc.tryexcept():
    fickle_func(n=1000)

sc.heading('Store all history')
tryexc = None
for i in range(1000):
    with sc.tryexcept(history=tryexc, verbose=False) as tryexc:
        fickle_func()
tryexc.disp()

————————————————————————————————————————————————
Simple usage, exit gracefully at first exception
————————————————————————————————————————————————

<class 'RuntimeError'> Value 0.993257 too big


—————————————————
Store all history
—————————————————

                      type                       value                             traceback
0   <class 'RuntimeError'>      Value 0.997336 too big  <traceback object at 0x7f905c117c80>
1   <class 'RuntimeError'>       Value 0.99658 too big  <traceback object at 0x7f9030653380>
2     <class 'ValueError'>  Value 0.00171662 too small  <traceback object at 0x7f9030653000>
3   <class 'RuntimeError'>       Value 0.99115 too big  <traceback object at 0x7f9030653540>
4   <class 'RuntimeError'>      Value 0.999809 too big  <traceback object at 0x7f9030653440>
5   <class 'RuntimeError'>      Value 0.991638 too big  <traceback object at 0x7f9030652840>
6   <class 'RuntimeError'>      Value 0.997487 too big  <traceback object at 0x7f9030653dc0>
7   <class 'RuntimeError'>      Value 0.998563 too big  <traceback object at 0x7f9030653740>
8   <class 'RuntimeError'>      Value 0.994992 too big  <traceback object at 0x7f90306521c0>
9   <class 'RuntimeError'>      Value 0.997475 too big  <traceback object at 0x7f9030651900>
10  <class 'RuntimeError'>      Value 0.999894 too big  <traceback object at 0x7f9030653fc0>

Interpolation and optimization

Sciris includes two algorithms that complement their SciPy relatives: interpolation and optimization.

Interpolation

The function sc.smoothinterp() smoothly interpolates between points but does not use spline interpolation; this makes it somewhat of a balance between numpy.interp() (which only interpolates linearly) and scipy.interpolate.interp1d(..., method='cubic'), which takes considerable liberties between data points:

import sciris as sc
import numpy as np
import pylab as pl
from scipy import interpolate

# Create the data
origy = np.array([0, 0.2, 0.1, 0.9, 0.7, 0.8, 0.95, 1])
origx = np.linspace(0, 1, len(origy))
newx  = np.linspace(0, 1)

# Create the interpolations
sc_y = sc.smoothinterp(newx, origx, origy, smoothness=5)
np_y = np.interp(newx, origx, origy)
si_y = interpolate.interp1d(origx, origy, 'cubic')(newx)

# Plot
kw = dict(lw=2, alpha=0.7)
pl.plot(newx, np_y, '--', label='NumPy', **kw)
pl.plot(newx, si_y, ':',  label='SciPy', **kw)
pl.plot(newx, sc_y, '-',  label='Sciris', **kw)
pl.scatter(origx, origy, s=50, c='k', label='Data')
pl.legend();

As you can see, sc.smoothinterp() gives a more “reasonable” approximation to the data, at the expense of not exactly passing through all the data points.

Optimization

Sciris includes a gradient descent optimization method, adaptive stochastic descent (ASD), that can outperform SciPy’s built-in optimization methods (such as simplex) for certain types of optimization problem. For example:

# Basic usage
import numpy as np
import sciris as sc
from scipy import optimize

# Very simple optimization problem -- set all numbers to 0
func = np.linalg.norm
x = [1, 2, 3]

with sc.timer('scipy.optimize()'):
    opt_scipy = optimize.minimize(func, x)

with sc.timer('sciris.asd()'):
    opt_sciris = sc.asd(func, x, verbose=False)

print(f'Scipy result:  {func(opt_scipy.x)}')
print(f'Sciris result: {func(opt_sciris.x)}')

scipy.optimize(): 13.3 ms
sciris.asd(): 4.38 ms
Scipy result:  4.829290232718364e-08
Sciris result: 2.897767167584095e-16

Compared to SciPy’s simplex algorithm, Sciris’ ASD algorithm was ≈3 times faster and found a result ≈8 orders of magnitude smaller.

Animation

And finally, let’s end on something fun. Sciris has an sc.animation() class with lots of options, but you can also just make a quick movie from a series of plots. For example, let’s make some lines dance:

pl.figure()
frames = [pl.plot(pl.cumsum(pl.randn(100))) for i in range(20)] # Create frames
sc.savemovie(frames, 'dancing_lines.gif'); # Save movie as a gif

Saving 20 frames at 10.0 fps and 150 dpi to "dancing_lines.gif" using imagemagick...
Done; movie saved to "dancing_lines.gif"
File size: 150 KB
Elapsed time: 6.06 s

This creates the following movie, which is a rather delightful way to end:

We hope you enjoyed this series of tutorials! Remember, write to us if you want to get in touch.