mcmc() runs warmup and sampling inside TensorFlow, returning to R only to update the progress bar:
hmc() and rwmh() inside TFverbose = FALSE, each phase, extra_samples()'s included, is one call to TensorFlow.mcmc() is about 2 times faster than in greta 0.6.0 for the first call on a model, and 2-3 times faster for each call after it. See (benchmarks) (#547, #834).mcmc() traces its sampler's loop once per model, sampler and number of chains, and reuses it on later calls, though each worker under a parallel future plan traces its own on every call (#547).
mcmc() and calculate() trace the model's log density once per model, rather than for each batch size. TensorFlow can still warn about retracing when a session builds several models, as reticulate wraps every function greta traces in the same Python code object (#546).
opt() with a Keras optimiser such as adam() runs inside one call to TensorFlow. This includes its whole loop, convergence and overflow checks. It also traces the log density once rather than twice, and keeps the loop on the model, so a later call with the same optimiser and settings will reuse it and start afresh from the initial values. On two of greta's example models, with 2000 iterations of adam(), opt() is 2.9-7.9 times faster than in greta 0.6.0 for the first call, and 29-47 times faster for each call after it (benchmarks) (#546, #547, #633).
opt(hessian = TRUE) builds the model's graph once for all targets rather than once per target, and uses TensorFlow's vectorised pfor, which traces afresh on every call, only for targets of at least 100 elements. On a model with twenty scalar targets, it is 6.3 times faster than in greta 0.6.0 for the first call on a model, and 16 times faster for each call after it (benchmarks). Small targets no longer trigger TensorFlow's retracing warning (#546).
mcmc() reproducible. It drew a seed from R's random numbers but never passed it to TensorFlow; now set.seed() and tensorflow::set_random_seed() give identical draws, whatever pb_update, verbose and one_by_one are, so seeded draws differ from greta 0.6.0 (#285, #427, #547).set.seed() and gave greta arrays created after the same seed the same names, so model() merged them into one variable (#366).calculate() with nsim and no seed uses up one of R's random numbers, as mcmc() does, rather than calling set.seed() (#285).adamax() and nadam() default to a learning_rate of 0.1, up from 0.001, which could not reach the optimum of even a five-parameter model in 2000 iterations. nadam()'s results also shift slightly, as Keras 3 implements Nadam differently (#633).hmc() draws its number of leapfrog steps afresh at every iteration. It drew one per return to R, so a whole burst, or all of sampling with verbose = FALSE, used one value, and a chain could stop mixing. With Lmin equal to Lmax it takes exactly that many steps, and it errors when Lmin or Lmax is below 1, or Lmin is above Lmax (#547, #745).mcmc() and extra_samples() run thin iterations per kept draw, not thin + 1, and one per warmup step, not two, so the same arguments run 1.4 to 1.5 times faster, where before they gave 1.6 to 4.9 times the effective samples (benchmarks). To keep a default run's effective samples, mcmc() defaults to warmup = 2000 and n_samples = 2000, and extra_samples() to n_samples = 2000, up from 1000; a script that sets these needs them doubled to match an earlier run (#318).|> (#810).lkj_correlation() and wishart() differently, though the distributions are unchanged (#633).install_tensorflow() is no longer re-exported as it skipped greta's version checks, so it could install a TensorFlow that greta then refuses; use install_greta_deps(), or tensorflow::install_tensorflow() (#638).as_data_mutable() declares data you will replace later, and data_values<- replaces them in place, so one model can be fitted to many datasets without being rebuilt or retraced. Mutable data cost more to evaluate than constant data, noticeably for a large dataset on several chains, so declare only the data you will replace. Their dimensions cannot change, and they print as greta array <mutable data>. See the "Fitting one model to many datasets" vignette (#181).greta_set_deps(greta_deps_spec(tf_version = "2.20.0", python_version = "3.9")); greta_sitrep() still accepts an existing Python 3.9 environment (#633).tf extra (tensorflow-probability[tf]), so TensorFlow Probability chooses the tf-keras it imports; greta's own pin made TensorFlow 2.17 and 2.18 impossible to install (#633).greta_deps_spec() checks the Python and TensorFlow Probability versions as well as TensorFlow's, and its error now names the supported range instead of failing after installing (#633, #638).GRETA_ALLOW_UNSUPPORTED_DEPS to "true", so an unsupported installation can be built on purpose to test how greta reports it; see vignette("installation") (#685).install_greta_deps() has improved error handling and now stops when a step fails, showing what failed, advice for failures greta recognises (pip's PIP_REQUIRE_VIRTUALENV, or a Python that TensorFlow has no release for) and where the logfile is. Previously it had less reliable methods, and would sometimes spuriously report "Python modules installed!". It also no longer breaks on install output containing braces (#663, #719).install_greta_deps() says when it is installing into an existing greta-env-tf2 conda environment, and if that fails, suggests reinstall_greta_deps(), since installing again reuses the broken environment (#684).open_greta_install_log() and write_greta_install_log() use the path in GRETA_INSTALLATION_LOG, rather than a file of that name in the working directory. The logfile is written when installation fails too, starts with a "Problems found" section of advice and error lines, and shows install output such as tensorflow<2.16 as text rather than HTML (#663, #713).greta-env-tf2 conda environment gets an error naming the installed and needed versions, rather than failing inside Keras. When greta cannot load from a conda environment, the error suggests greta_remove("env") and a restart to switch to greta's managed environment, or reinstall_greta_deps() to rebuild the conda one (#638, #684).[<- on a greta array fills a matrix or array replacement column by column, as R does (#844).mcmc() and extra_samples() keep n_samples %/% thin draws, regardless of values of pb_update and one_by_one. A burst shorter than thin used to error, as in a final partial burst, one_by_one = TRUE with thin above 1, or extra_samples() with pb_update at or below thin. They run all n_samples iterations even when thin does not divide it, one_by_one = TRUE rejects only the proposal that caused a numerical error, and a thin larger than n_samples is an informative error (#241, #318, #567, #609).mcmc()'s progress bar reaches its total when pb_update does not divide the number of iterations, and fills in step with the iterations run; a bar could read 1010/1010 while still empty (#318).opt()'s convergence comes from the optimiser: bfgs() and nelder_mead() report it, and a Keras optimiser such as adam() has converged when its last step changed the objective by no more than tolerance, rather than being guessed from whether it stopped before max_iterations - 1 (#569).greta_deps_tf_tfp dataset is deprecated and will be removed. It is a fixed snapshot that stops at TensorFlow 2.15.1, so it gives the wrong answer about which versions work; greta instead tests a range of TensorFlow versions weekly on Linux, macOS and Windows (#633).DESCRIPTION has no Collate field: the code that builds greta_stash and .internals runs when the package loads, so the order of R/ no longer matters. .internals is unchanged, so greta.gp, greta.dynamics and greta.gam need no change (#457).TF_NUM_INTRAOP_THREADS and TF_NUM_INTEROP_THREADS) so CRAN's CPU/elapsed timing on vignette rebuilds stays under the two-core limit (#796).R CMD check (detected via the _R_CHECK_LIMIT_CORES_ environment variable), so checks on CRAN machines respect the two-core limit (#796).dim<- now keeps the ? placeholder display for unknown values instead of showing NA (#582).== 2.16.0 and now accepts >= 2.16.0.n_cores argument now defaults to 2 cores and chains defaults to 2 chains; when the number of cores requested exceeds the number detected, the number detected is used..batch_size instead of batch_size internally, to avoid rare name clash errors (#634).as.unknowns() now handles plain numeric vectors (and dim<- being set to NULL), fixing an mcmc() error "no applicable method for 'as.unknowns'" that surfaced when re-building vignettes under R-devel (#582).greta_notes_tf_num_error() now emits its note with message() rather than cat(), so it can be silenced with suppressMessages() and is written to the message stream.greta_sitrep() gains a verbosity argument with three levels of detail: "minimal" (default), "detailed", and "quiet" (#612, resolved by #679).log.greta_array() now warns when the base argument is supplied, as it was being ignored (#597).outer() (and %o%) now works with greta arrays when FUN = "*", instead of silently returning a base array of NAs; base R's "*" fast path used as.vector(), which dropped the greta operation (#582).RETICULATE_PYTHON, a stored greta preference, or an existing greta-env-tf2 conda environment, and otherwise uses the managed (uv) environment (#801).UV_OFFLINE) once reticulate's uv cache is installed. You can set UV_OFFLINE=0 (or =1) yourself to force online or offline resolution, respectively (#814).library(greta) failing with an error when greta's stored Python preference file exists but is empty (#809).greta_deps_spec() now only checks that the requested TensorFlow version is one greta supports (TensorFlow 2.16 and later are not supported, as they ship Keras 3); compatible TensorFlow Probability and Python versions are left to uv or conda to resolve rather than being validated against a fixed compatibility table (#675).greta_list_py_modules() shows the Python packages installed in a specific TF2 environment (#801, #809).greta_remove() consolidates greta's Python removal helpers behind a single what argument to remove the greta-env-tf2 conda environment, miniconda, reticulate's uv cache, the stored Python preference, the stored dependency versions, or all of them. This means destroy_greta_deps(), greta_remove_all_deps(), remove_greta_env(), remove_miniconda(), and remove_reticulate_uv_cache() are now superseded by greta_remove() (#814).greta_remove() no longer leaves a stale, deleted environment active for the rest of the session: it now invalidates greta's cached Python backend and nudges you to restart R if you try to use greta again without restarting, instead of silently failing or falsely reporting the removed environment as still available.greta_set_deps() persistently chooses which TensorFlow, TensorFlow Probability, and Python versions greta uses; the managed (uv) environment installs them on next load and install_greta_deps() uses them as its default; clear with greta_remove("deps") (#817).greta_set_python() and greta_reset_python() let you choose, persistently, which Python environment greta uses - the managed (uv) environment (backend = "uv", the default), a conda environment (backend = "conda"), or a specific Python (backend = "path"); each reports the stored preference, warns if RETICULATE_PYTHON takes precedence, and shows what greta will resolve to on its next load (#801, #809, #817).greta_set_python("path", path = ...) accepts either a Python binary or an environment directory (a virtualenv or conda prefix), looking for bin/python (or Scripts/python.exe on Windows) inside it, which eases offline and pre-installed setups (#814).greta_sitrep() now reports the resolved Python backend and whether greta can start offline: for the managed (uv) environment, the UV_OFFLINE setting and whether reticulate's uv cache is populated; a missing conda environment is reported neutrally as "not used" on the managed (uv) backend rather than as a failure (#801, #817).greta_sitrep() now requires Python 3.9 or later (previously 3.8), matching the Python versions greta supports (#809).install_greta_deps() now records the location of the greta-env-tf2 conda environment at install time, so greta auto-detects it in any conda installation, not just reticulate's miniconda (#809).install_greta_deps() is no longer required for most users, as greta now installs TensorFlow and TensorFlow Probability automatically via uv on first use; it remains for installing a conda environment (for example, for offline use), which you can then select with greta_set_python("conda") (#801).install_greta_deps() now restores the warning.length option when it exits, instead of leaving its own value in place for the rest of the session.reinstall_greta_env() and reinstall_miniconda() are now deprecated in favour of reinstall_greta_deps() (#814).remove_greta_env(), remove_miniconda(), and remove_reticulate_uv_cache() ask for confirmation before removing, gain an ask argument (default interactive()) so they work non-interactively, and invisibly return whether anything was removed (#809).remove_reticulate_uv_cache() removes reticulate's uv cache; note this cache is shared by all reticulate packages and is not greta-specific, and a system-wide uv cache is left untouched (#801, #809).This version of greta uses Tensorflow 2.0.0, which comes with it a host of new very exciting features!
The latest interface to optimizers in tensorflow are now used, these changes are described.
gradient_descent gains momentum and nesterov arguments, as described here in TF docsadagrad gains epsilon argumentmomentum optimizer, as this has been folded into gradient_descent argumentsamsgrad argument to adam optimizer, as described in TF docsadamax optimiser, see TF docsl2_shrinkage_regularization_strength and beta arguments to ftrl
optimiser.nadam optimiser - see docs.rms_prop optimiser, changes decay parameter to rho, and adds centered parameter - see docsThe following optimisers are removed, as they are no longer supported by Tensorflow:
powell()cg()newton_cg()l_bfgs_b()tnc()cobyla()slsqp()This release provides a few improvements to installation in greta. It should now provide more information about installation progress, and be more robust. The intention is, it should just work, and if it doesn't, it should fail gracefully with some useful advice on problem solving.
library(greta) after installation (#523).greta_deps_spec() to help simplify specifying package versions (#664).method and conda arguments from install_greta_deps() as they
were not used.manual argument in install_greta_deps().greta_deps_tf_tfp (#666), which contains valid versions combinations of TF, TFP, and Python.greta_nodes_install/conda_*() options as #493 makes them defunct.greta_set_install_logfile(), and write_greta_install_log(), and open_greta_install_log() (#493).destroy_greta_deps() function to remove miniconda and python conda environment.write_greta_install_log() and open_greta_install_log() to use tools::R_user_dir() to always write to a file location. open_greta_install_log() will open one found from an environment variable or go to the default location (#703).greta_deps_receipt() records the TensorFlow, TensorFlow Probability, and Python versions that are actually installed without validating them against greta's supported range, so it works even when the installed versions are newer than greta officially supports (such as a TensorFlow 2.15 patch release) (#668).greta_mcmc_list. This means MCMC output will be shorter and more informative (#644).n argument: print(object, n = <elements to print>) (#644).greta_sitrep() now checks for installations of Python, TF, and TFP.is.greta_array() and is.greta_mcmc_list().restart argument for install_greta_deps() and reinstall_greta_deps() to automatically restart R (#523).check_* functions.cli::cli_abort/warn/inform() in place of cli::format_error/warning/message() + stop/warning/message(msg, call. = FALSE) pattern.tf$keras$optimizers$legacy$METHOD over tf$keras$optimizers$METHOD). No user impact expected.%||% internally to replace the pattern: if (is.null(x)) x <- thing with x <- x %||% thing (#630).if (thing & thing & what == this) with if (explanation_of_thing).vapply into functions (#377, #658)..deps_tf and .deps_tfp to track dependencies of TF and TFP. Related to #666.forward_log_det_jacobian() function from tf_correlation_cholesky_bijector() (used in lkj_correlation()). Previously, it did not work with unknown dimensions, but it now works with them.tf$matmul(chol_draws, chol_draws, adjoint_b = TRUE) instead of tf_chol2symm(chol_draws).FillTriangular with FillScaleTriL and apply Chaining to first transpose input.reinstall_greta_deps(), which helps with starting from a clean slate when installing greta dependencies (#524)future and parallely packages error when a CPU with only one core is provided (#513, #516).multiprocess as it is deprecated in the future package (#394)Python is now initialised when a greta_array is created (#468).
head and tail S3 methods for greta_array are now consistent with head and tail methods for R versions 3 and 4 (#384).
greta_mcmc_list objects (returned by mcmc()) are now no longer modified by operations (like coda::gelman.diag()).
joint distributions of uniform variables now have the correct constraints when sampling (#377).
array-scalar dispatch with 3D arrays is now less buggy (#298).
greta now provides R versions of all of R's primitive functions (I think), to prevent them from silently not executing (#317).
Uses Sys.unsetenv("RETICULATE_PYTHON") in .onload on package startup,
to prevent an issue introduced with the "ghost orchid" version of RStudio where they do not find the current version of RStudio. See #444 for more details.
Internal change to code to ensure future continues to support parallelisation of chains. See #447 for more details.
greta now depends on future version 1.22.1, tensorflow (the R package) 2.7.0, and parallelly 1.29.0. This should see no changes on the user side.
Now depends on R >= 3.1.0 (#386)
chol2inv.greta_array() now warns user about LINPACK argument being ignored, and also reminds user it has been deprecated since R 3.1
calculate() now accepts multiple greta arrays for which to calculate values, via the ... argument. As a consequence any other arguments must now be named.
A number of optimiser methods are now deprecated, since they will be unavailable when greta moves to using TensorFlow v2.0: powell(), cg(), newton_cg(), l_bfgs_b(), tnc(), cobyla(), and slsqp().
dirichlet() now returns a variable (rather than an operation) greta array, and the graphs created by lkj_correlation() and wishart() are now simpler as cholesky-shaped variables are now available internally.
Adds the reinstall_greta_env(), reinstall_miniconda(), remove_greta_env(), and remove_miniconda() helper functions for helping installation get to "clean slate" (#443).
greta currently doesn't work on Apple Silicon (M1 Macs) as they need to use TF 2.0, which is currently being implemented. greta now throws an error if M1 macs are detected and directs users to https://github.com/greta-dev/greta/issues/458 (#487)
New install_greta_deps() - provides installation of python dependencies (#417). This saves exact versions of Python (3.7), and the python modules NumPy (1.16.4), Tensorflow (1.14.0), and Tensorflow Probability (0.7.0) into a conda environment, "greta-env". When initialising Python, greta now searches for this conda environment first, which presents a great advantage as it isolates these exact versions of these modules from other Python installations. It is not required to use the conda environment, "greta-env". Overall this means that users can run the function install_greta_deps(), follow the prompts, and have all the python modules they need installed, without contaminating other software that use different python modules.
calculate() now enables simulation of greta array values from their priors, optionally conditioned on fixed values or posterior samples. This enables prior and posterior predictive checking of models, and simulation of data.
A simulate() method for greta models is now also provided, to simulate the values of all greta arrays in a model from their priors.
variable() now accepts arrays for upper and lower, enabling users to define variables with different constraints.
There are three new variable constructor functions: cholesky_variable(), simplex_variable(), and ordered_variable(), for variables with these constraints but no probability distribution.
New chol2symm() is the inverse of chol().
mcmc(), stashed_samples(), and calculate() now return objects of class greta_mcmc_list which inherit from coda's mcmc.list class, but enable custom greta methods for manipulating mcmc outputs, including a window() function.
mcmc() and calculate() now have a trace_batch_size argument enabling users to trade-off computation speed versus memory requirements when calculating posterior samples for target greta arrays (#236).
Many message, warning, and error prompts have been replaced internally with the {cli} R package for nicer printing. This is a minor change that should result in a more pleasant user experience (#423 #425).
Internally, where sensible, greta now uses the glue package to create messages/ouputs (#378).
New FAQ page and updated installation instructions for installing Python dependencies (#424)
New greta_sitrep() function to generate a situation report of the software
that is available for use, and also initialising python so greta is ready to
use. (#441)
This release is predominantly a patch to make greta work with recent versions of TensorFlow and TensorFlow Probability, which were not backward compatible with the versions on which greta previously depended. From this release forward, greta will depend on specific (rather than minimum) versions of these two pieces of software to avoid it breaking if more changes are made to the APIS of these packages.
greta now (only) works with TensorFlow 1.14.0 and TensorFlow Probability 0.7.0 (#289, #290)
behaviour of the pb_update argument to mcmc() has been changed slightly to avoid a bad interaction with thinning (#284)
various edits to the documentation to fix spelling mistakes and typos
This is a very large update which adds a number of features and major speed improvements. We now depend on the TensorFlow Probability Python package, and use functionality in that package wherever possible. Sampling a simple model now takes ~10s, rather than ~2m (>10x speedup).
dim<-() now always rearranges elements in column-major order (R-style, not Python-style)future package for execution of MCMC chains on remote machines. Note: it is not advised to use future for parallel execution of chains on the same machine, that is now automatically handled by greta.one_by_one argument to MCMC can handle serious numerical errors (such as failed matrix inversions) as 'bad' samplesextra_samples() function to continue sampling from a model.calculate() works on the output of MCMC, to enable post-hoc posterior predictionmixture() and joint() distribution constructorsabind(), aperm(), apply(), chol2inv(), cov2cor(), eigen(), identity(), kronecker(), rdist(), and tapply() (thanks to @jdyen)greta_array()opt() and mcmc() as objects, with defined tuning parameters. The control argument to these functions is now defunct.x[2, 3], rather than x.6)plot.greta_model() now returns a DiagrammeR::grViz object (thanks to @flyaflya). This is less modifiable, but renders the plot more much consistently across different environments and notebook types. The DiagrammeR dgr_graph object use to create the grViz object is included as an attribute of this object, named "dgr_graph".Minor patch to handle an API change in the progress package. No changes in functionality.
model(), %*%<- for assignmentn_cores argument to model()calculate() function to compute the values of greta arrays conditional on provided values for othersimultilogit() transformchains argument to model()cumsum() and cumprod() functionsforwardsolve() and backsolve()colSums(), rowSums(), colMeans(), and rowMeans()dim<-() to reshape greta arrayssweep() now handles greta array STATS when x is numeric.internals object to enable extension packagesAPI changes:
define_model(), an alias for model()