Developing manuscripts w/ embedded notebooks

[jmlondon]

Description

Hello

I am developing a scientific manuscript for eventual submission to a peer-reviewed journal. For this, I've adopted the 'manuscript' project type. From past experience with Rmarkdown/bookdown projects, I have an appreciation for separating the content/code into separate files as opposed to a single very large rmd/qmd file. To facilitate this, I've adopted the use of additional notebook *.qmd files to handle generation of specific figures. And, I've created a content directory that has introduction.qmd, methods.qmd, results.qmd etc.

So, my directory structure looks something like the following:

my_manuscript/
  |- _quarto.yml
  |- index.qmd
  |- references.yaml
  |- apa.csl
  |- content/
      |- introduction.qmd
      |- methods.qmd
      |- results.qmd
  |- notebooks/
      |- figure_map.qmd
      |- figure_chart.qmd

For my _quarto.yml file, I have the following specified:

project:
  type: manuscript

execute:
  freeze: false
  echo: false
  warning: false
  message: false

bibliography: references.yaml
csl: apa.csl

format:
  html:
    toc: true

Within the index.qmd file I have all of the typical scholarly writing YAML frontmatter (e.g. title, author, affiliatons, abstract, etc) and then just the following to specify the content includes

{{< include content/introduction.qmd >}}

{{< include content/methods.qmd >}}

{{< include content/results.qmd >}}

Within those content files, I'll take a similar approach to include embeds from other notebooks for my figures. For example

# Methods

## Study area and species

Lorem ipsum dolor sit amet consectetur adipiscing elit. Dolor sit 
amet consectetur adipiscing elit quisque faucibus.

## Location data

{{< embed /notebooks/figure_map.qmd#fig-my-map >}}

With that background on my setup and reasoning, here are a few things I've noticed while developing and some questions for advice and best practices moving forward. I'll note that I'm largely using Positron these days but in discussions with a colleague using RStudio, I think most of these observations and questions are relevant to both IDEs. Also, I realize there's a chance some of my questions and observations below are more relevant to the IDE than Quarto. If so, I'll be happy to move those topics over to the Positron discussions.

My Questions & Observations:

1. In this situation (with notebooks for include and embed content), how/when is the Preview updated? I really find the combination of Quarto with the Positron Visual Editor Pane and Preview Pane side-by-side (and Render on Save checked) a very satisfying and enjoyable manuscript editing experience. But, it seems to me that Preview does not update the full manuscript site when changes are made to any of the include/embed notebooks. For example, if I Preview index.qmd and then make and edit to methods.qmd as soon as I save methods.qmd, the methods notebook re-renders. But, the full project does not. So, I have to jump over to index.qmd and re-render/preview. Is this expected behavior? Is there a way to trigger a preview update when dependent files are updated? It might be helpful if I had the option to configure quarto such that running quarto preview would default to rendering the full project and rely on quarto preview methods.qmd to preview a specific file.
2. Is there a functional difference between Preview and Render in this situation that's important for me to understand? As I've been developing the manuscript with this workflow, I sometimes notice that a particular figure doesn't update with the preview or, maybe, the figure updates but is at a lower resolution that I've specified. I can usually resolve this by stopping the Preview background server and rendering index.qmd from the command palette. Then, if I run Preview again, the content is fully updated. Preview doesn't always seem so good at cleaning up extra files. So, I am regularly initiating a full render to clean things up (especially before committing to version control)
3. How should I be using freeze in this scenario? You'll notice I've set freeze to false within _quarto.yml. This is largely because I don't fully understand how it might be impacting the output of preview/render in this situation and I was suspicious it might be the cause for some of the times when content wasn't fully updating. Ideally, freeze would be aware of changes to either the main article file or any dependent files and cache those results so the full site doesn't have to update each time a small piece is updated.
4. I think the ability to set global parameters for embed content could be useful In this scenario I've described, I may have 8-10 separate notebooks for figures and I need to specify specifics like fig-width, dpi, etc. for each. Often, these need to be configured for each specific figure. But, there are common settings and having a single place to the set them would be convenient.
5. Is it possible for the parent R environment from index.qmd to propagate to the other notebooks? I think I know the answer to this and that each notebook is fully independent. I'm specifically thinking about this with respect to loading libraries or working with common data objects. It seems I need to repeat calls to load libraries/objects for each notebook. I was hoping, for instance, that I could have an R code chunk within index.qmd that loads all the libraries needed in a central place. Similarly, such an approach would be useful for setting ggplot themes or other variables.

Thank you for your time both in the development of Quarto but also to answer these questions and help me wrap my head around the optimal workflow for manuscripts. I've spent quite a bit of time reading through the documentation and it has been a great resource. These are questions I just couldn't fully grasp from the documentation so I appreciate having the opportunity for further engagement.

[mcanouil]

1. quarto preview will try to detect the resources being used by the document you are previewing. Unfortunately, it's not perfect, so a new preview might not be triggered when embedded/included documents or other resources changes. I won't list all the issues: https://github.com/quarto-dev/quarto-cli/issues?q=state%3Aopen%20label%3Apreview
2. quarto preview does quarto render but only on the subset of the project to allow the preview of the document on which you executed the preview. Preview mostly does incremental rendering. quarto render does complete rendering of the project unless you specify an input document.
3. freeze is generated on the first render and every time you specifically render/preview a document. There are flags/options to force (or the opposite) the use of the freeze. See the documentation.
4. and 5. I suggest you search the discussions because there are already several threads on the topic with several solutions.