---
title: "magclass 6"
author: "Jan Philipp Dietrich"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{magclass 6}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, echo = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

The step from magclass 5.29 to 6.00 marks a major rewrite of the whole package. While the definition of the magclass object has been refined and generalized over time (allowing multiple sub-dimensions in all main dimensions, introduction of set names, etc.) some functions of the package still relied on implicit assumptions which were not necessarily valid anymore for the new definitions. This led to situations in which functions might only work for specific cases (such as functions for sub dimension manipulation only working in the third dimension) or even return wrong results in some instances. 

This tutorial will go through the major changes coming with version 6 and what they mean for working with magclass objects.

## Testing

Magclass comes now with more than 600 unit tests which are covering more than 90% lines of code of the total package. These tests should help to detect code breaking changes already early in the process and reduce the risk of pushing incompatible modifications to our main repository. 
In addition, it is planned to expand the test collection with every new feature or newly observed bug so that if issues arise that they only arise once. 

## (soon to be) deprecated functions

There are some function in magclass building on the old standard of having exactly one spatial and one temporal dimension (no subdimensions), which has been generalized in the meantime (up to 9 subdimension in every of the three main dimensions allowed). So far only two of them received a deprecation warning, which are `fulldim` (does not split subdimensions in first two dimensions) and `getRegionList` (assumes to only have regions in the first dimension). 
Functionality can now be found in the new function `getItems` which allows to return elements of (sub-)dimensions in various ways:

```{r, echo = TRUE}
library(magclass)
a <- maxample("animal")
p <- maxample("pop")
```

instead of `fulldim(a)` to receive a list of all sub-dimensions use now:

```{r, echo = TRUE}
getItems(a, split = TRUE)
```


In contrast to `fulldim` this splits all dimensions in sub-dimensions and reports the corresponding set names.

It is also possible to split a specific main dimension into sub-dimensions:

```{r, echo = TRUE}
getItems(a, dim = 1, split = TRUE)
```

Instead of `getRegionList(a)` one should use `getItems` as well:

```{r, echo = TRUE}
getItems(a, dim = 1.1, full = TRUE)
```

This command gives the same results as `getRegionList` but makes it more obvious what the command is doing (returning the first spatial sub-dimension in full length)

Some other functions such as `getRegions` or `getCells` have not been formally deprecated due to that broad usage, but are encouraged to be replaced as well with `getItems`. 

Another function which recieved a deprecation warning is `write.report2`. So far the package contained two very similar functions `write.report` and `write.report2` from which the latter is the more efficient one. In the new version the old `write.report` function has been replaced with the code of `write.report2` so that in the future all computation are based on the same function. 

## New feature: extended sub-dimension support

So far `dimSums`, `add_columns`, `add_dimension` were only functioning for the sub-dimensions in the third dimension while `getCPR` were only functioning for the first dimension. All these functions have been completely rewritten and work now for all sub-dimensions and are able to address a dimension either by its `dimCode` or set name:

```{r, echo = TRUE}
dimSums(a, c("x", "y", "cell", "day", "month", "species", "color"))

getCPR(a, 3.2)

add_dimension(p[, 1, 1], 1.2)

add_columns(p[, 1, 1], dim = "i")
```
## New magclass subsetting variant

To ease the process to write generalized functions which work for all main dimensions a new sub-setting option has been introduced which allows to specify the main dimension in which the sub-setting should take place via a new `dim` argument:

```{r, echo = TRUE}
p[3, dim = 2] # equivalent to p[,3,]
```
## Generalized handling of spatial raster data

Previously spatial data could only be handled for a pre-defined set of 59199 0.5x0.5 degree cells for which cells were numbered from 1 to 59199 and coordinates had to be known by the user and properly assigned to these cells. 
With the update raster data instead gets assigned its coordinates in the spatial dimension allowing also to handle data for differing resolutions and/or coverage. Accordingly, `read.magpie` and `write.magpie` have been adapted to this new behavior and custom implemetations for `ncdf4` and `asc` file support have been replaced with the read/write routines provided by the `raster` package. 
For applications this means that the behavior of code reading/writing these files might change and might require some modifications, but for the future it should ease the handling and exchange of spatial data with magclass objects.

## One-stop-shop-function getItems

As indicated earlier `getItems` can be used to replace `fulldim` and `getRegionList`. But its functionality goes beyond it making it the function to be used when elements of (sub-)dimensions should be read, added, removed or modified. While so far many different functions had to be remembered for these applications (such as `getCells`, `getNames`, `add_dimensions`) it is now recommended to first look at `getItems`. In the following some examples of its functionality:

Returning a sub-dimension:

```{r, echo = TRUE}
getItems(a, dim = 3.2)
getItems(a, dim = "color")
```

Returning the full vector of a sub-dimension:

```{r, echo = TRUE}
getItems(a, dim = 3.2, full = TRUE)
```

Returning a full main dimension...

```{r, echo = TRUE}
getItems(a, dim = 3)
```

...split into its sub-dimensions

```{r, echo = TRUE}
getItems(a, dim = 3, split = TRUE)
```

Replace elements element-wise:

```{r, echo = TRUE}
getItems(a, dim = "color") <- paste0("color", 1:4)
getItems(a, dim = 3)
```

Replace the whole vector:

```{r, echo = TRUE}
getItems(a, dim = "color", full = TRUE) <- paste0("color", 1:5)
getItems(a, dim = 3)
```

Delete a dimension:

```{r, echo = TRUE}
getItems(a, dim = "color") <- NULL
getItems(a, dim = 3)
```

Add a dimension:

```{r, echo = TRUE}
getItems(a, maindim = 3, dim = "newcolor") <- paste0("color", 1:5)
getItems(a, dim = 3)
```

All these manipulations work for all (sub-)dimensions of the object.

## Internal code modifications

As the complete package underwent a code review also many other functions which have not been mentioned here received modifications. While this should affect the behavior of these functions it improved in some cases the efficiency of the underlying computations.