Skip to content

How to configure B2SHARE#

All file paths mentioned are in inveniordm_deployment folder.

Configuration modules#

B2SHARE configuration is split to multiple configuration modules found in site/b3share/configurations/. The module split is done to reduce the amount of configuration in invenio.cfg file. Each configuration module has at least a dict called CONFIG. The modules are designed to have default values and the modules should not be touched during set up of a new B2SHARE instance.

The structure of configuration module is:

# imports

CONFIG = {
    "key": "value"
}

REQUIRED = ["list of required config variables"]

def required_keys(config):
    return REQUIRED # if config.get("key") else []

...

Each new configuration module has to be added to site/b3share/configurations/__init__.py and invenio.cfg. To __init__.py configuration modules have to be imported, added to __all__ and if the module has any logic added to B3SHARE_CONFIG_MODULES. To invenio.cfg a configuration module has to be imported and a line apply_config(globals(), <module>.CONFIG) added.

invenio.cfg#

We aim to keep invenio.cfg file as clean as possible. To reduce clutter default configurations are handled with configuration modules (check previous chapter) and instance specific configuration should be in env files and/or in vault. For development and testing invenio.cfg can be used to host configuration variables.

Env files#

B2SHARE has two env files: .env and app.env. app.env is used to host all application specific configuration variables. .env is used to host all variables required for docker compose deployment. Configuration variables in app.env have to be prefixed with INVENIO_. There are examples of both of these files (b2share.env => .env, b2share.app.env => app.env) in the B2SHARE repo.

Vault integration#

B2SHARE supports Vault for secrets management. To use Vault the B2SHARE deployment requires vault-agent container:

vault-agent:
    image: vault-agent:1.20.2
    # Do not do UID GID mapping for vault-agent.
    restart: "unless-stopped"
    # Do not use ports: since this does not need to be exposed to the host. Only to docker stack.
    expose:
        - "8100"
    env_file: "agent_dev.env"
    healthcheck:
        test: ["CMD", "find", "/pidfile"]
        interval: 5s
        timeout: 5s
        retries: 3
        start_period: 5s

The vault-agent container needs it’s own configuration variables hosted in agent_dev.env. Required configuration for vault-agent to run are:

AGENT_ROLE_ID=
AGENT_SECRET_ID=
VAULT_ADDR=
APP_ROLE=
APP_ROLE_ID=
FORCE_DEBUG=(not required, True/False)

To enable Vault integration for B2SHARE the following configuration variables have to be in app.env:

INVENIO_CONFIG_USE_VAULT_AGENT=True/False
INVENIO_CONFIG_VAULT_ENTITY=
INVENIO_CONFIG_VAULT_AGENT_ADDR="http://agent:8100"

List of available B2SHARE configuration variables#

Configuration variables with default values.

Base configuration#

SEND_FILE_MAX_AGE_DEFAULT=300
APP_DEFAULT_SECURE_HEADERS={
    "content_security_policy": {
        "default-src": [
            "'self'",
            "data:",
            "'unsafe-inline'",
            "blob:",
        ],
    },
    "content_security_policy_report_only": False,
    "content_security_policy_report_uri": None,
    "force_file_save": False,
    "force_https": True,
    "force_https_permanent": True,
    "frame_options": "sameorigin",
    "frame_options_allow_from": None,
    "session_cookie_http_only": True,
    "session_cookie_secure": True,
    "strict_transport_security": True,
    "strict_transport_security_include_subdomains": True,
    "strict_transport_security_max_age": 31556926,
    "strict_transport_security_preload": False,
}
FLASK_DEBUG=1
LOGGING_CONSOLE=True
LOGGING_CONSOLE_LEVEL="DEBUG"
LOGGING_FS_LEVEL="INFO" # File logging
LOGGING_FS_LOGFILE="/opt/invenio/var/instance/archive/invenio.log" # Log file path
ADMINISTRATION_DISPLAY_VERSIONS=[
    ("invenio-app-rdm", f"v{invenio_version}"),
    ("b3share", f"v{b3version}"),
]
VERSION=b3version # Imported from b3share __version__
SITE_UI_URL=None # REQUIRED
SITE_API_URL=None # REQUIRED
JSONSCHEMAS_REGISTER_ENDPOINTS_API=True
JSONSCHEMAS_REGISTER_ENDPOINTS_UI=True
MIGRATION=False # DANGEROUS! Skips some permission checks when enabled

Celery (job scheduling) configuration#

CELERY_BEAT_SCHEDULE={
    **CELERY_BEAT_SCHEDULE,
    "storage-daily-rollup": {
        "task": "b3share.statistics.tasks.stat_storage_daily",
        "schedule": crontab(hour=1, minute=0),
    },
    "check-remote-file-reachability": {
        "task": "b3share.remote_files.tasks.check_remote_file_reachability",
        "schedule": crontab(hour=2, minute=0),
    },
}

Database configuration#

SQLALCHEMY_ENGINE_OPTIONS={
    "pool_pre_ping": False,
    "pool_recycle": 3600,
    # set a more agressive timeout to ensure http requests don't wait for long
    "pool_timeout": 5,
}
POSTGRES_USER=None # REQUIRED
POSTGRES_PASSWORD=None # REQUIRED
POSTGRES_DB=None # REQUIRED

OAuth configuration#

# Local login
ACCOUNTS_LOCAL_LOGIN_ENABLED=False
SECURITY_REGISTERABLE=False
SECURITY_RECOVERABLE=False
SECURITY_CHANGEABLE=False
SECURITY_CONFIRMABLE=False
SECURITY_LOGIN_WITHOUT_CONFIRMATION=False

# Signup form
OAUTHCLIENT_SIGNUP_FORM=_create_registrationform
ACCOUNTS_LOGIN_VIEW_FUNCTION=auto_redirect_login
OAUTHCLIENT_AUTO_REDIRECT_TO_EXTERNAL_LOGIN=False

# Oauth clients
B2ACCESS_ENABLED=False
CSCAAI_ENABLED=False
B2ACCESS_CONSUMER_KEY=None # Required if B2ACCESS_ENABLED
B2ACCESS_SECRET_KEY=None # Required if B2ACCESS_ENABLED
B2ACCESS_BASE_URL=None # Required if B2ACCESS_ENABLED
CSCAAI_CONSUMER_KEY=None # Required if CSCAAI_ENABLED
CSCAAI_SECRET_KEY=None # Required if CSCAAI_ENABLED
CSCAAI_BASE_URL=None # Required if CSCAAI_ENABLED

OAUTH_ALLOWED_ORGANIZATIONS=[] # Limit login to specific domains only

PID configuration#

# DataCite DOI
DATACITE_ENABLED=False
DATACITE_PREFIX=None
DATACITE_FORMAT="{prefix}/b2share.{id}"
DATACITE_TEST_MODE=None
DATACITE_USERNAME=None
DATACITE_PASSWORD=None
DATACITE_DATACENTER_SYMBOL=""

# Handle PID
HANDLE_ENABLED=False
HANDLE_PREFIX=None
HANDLE_FAKE=None
HANDLE_SYSTEM_BASEURL=None
HANDLE_PID_HANDLE_CREDENTIALS={
  "handle_server_url": "",
  "private_key": "",
  "certificate_only": "",
  "prefix": "",
  "handleowner": "",
  "reverse_username": "",
  "reverse_password": "",
  "HTTPS_verify": "True",
  "client": "rest"
}

# Badges
FORMATTER_BADGES_ALLOWED_TITLES=["DOI", "EPIC"]
FORMATTER_BADGES_TITLE_MAPPING={"doi": "DOI", "epic": "EPIC"}

# General
RDM_PERSISTENT_IDENTIFIER_PROVIDERS=CUSTOM_RDM_PERSISTENT_IDENTIFIER_PROVIDERS
RDM_RECORDS_IDENTIFIERS_SCHEMES=CUSTOM_RDM_RECORDS_IDENTIFIERS_SCHEMES
RDM_PERSISTENT_IDENTIFIERS=CUSTOM_RDM_PERSISTENT_IDENTIFIERS
RDM_ALLOW_EXTERNAL_DOI_VERSIONING=False
CUSTOM_RDM_PERSISTENT_IDENTIFIERS={
    "doi": {
        "providers": ["datacite"],
        "required": True,
        "label": lazy_("DOI"),
        "validator": idutils.is_doi,
        "normalizer": idutils.normalize_doi,
        "is_enabled": providers.DataCitePIDProvider.is_enabled,
        "ui": {"default_selected": "no"},
    },
    "oai": {
        "providers": ["oai"],
        "required": True,
        "label": lazy_("OAI"),
        "is_enabled": providers.OAIPIDProvider.is_enabled,
    },
    "epic": {
        "providers": ["epic"],
        "required": True,
        "label": lazy_("epic"),
        "is_enabled": PyhandlePIDProvider.is_enabled,
    },
    "b2rec": {
        "providers": ["b2rec"],
        "required": False,
        "label": lazy_("b2rec"),
        "is_enabled": B2RECPIDProvider.is_enabled,
    },
}
CUSTOM_RDM_PERSISTENT_IDENTIFIER_PROVIDERS=[
    *_upstream_providers,
    PyhandlePIDProvider("epic", label=lazy_("epic")),
    B2RECPIDProvider("b2rec", label=lazy_("b2rec")),
]

CUSTOM_RDM_RECORDS_IDENTIFIERS_SCHEMES={
    **RDM_RECORDS_IDENTIFIERS_SCHEMES,
    "b2rec": {
        "label": "B2SHARE Legacy Record ID",
        "validator": validate_b2rec,
    }
}

Mail configuration#

MAIL_SUPPRESS_SEND=None
MAIL_SERVER=None
MAIL_PORT=None
MAIL_USE_TLS=False
MAIL_USE_SSL=False
SECURITY_EMAIL_SENDER=None
MAIL_DEFAULT_SENDER=None
MAIL_REPLY_TO=None
APP_RDM_ADMIN_EMAIL_RECIPIENT=None
NOTIFICATIONS_BACKENDS={
    **NOTIFICATIONS_BACKENDS,
    AbuseReportBackend.id: AbuseReportBackend(),
}
NOTIFICATIONS_BUILDERS={
    **NOTIFICATIONS_BUILDERS,
    AbuseReportNotificationBuilder.type: AbuseReportNotificationBuilder,
}

Record specific configuration#

RDM_ALLOW_METADATA_ONLY_RECORDS=True
RDM_ALLOW_RESTRICTED_RECORDS=True
APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES="search"

APP_RDM_DEPOSIT_FORM_DEFAULTS={
    "publication_date": lambda: datetime.now().strftime("%Y-%m-%d"),
    "rights": [
        {
            "id": "cc-by-4.0",
            "title": "Creative Commons Attribution 4.0 International",
            "description": (
                "The Creative Commons Attribution license allows "
                "re-distribution and re-use of a licensed work "
                "on the condition that the creator is "
                "appropriately credited."
            ),
            "link": "https://creativecommons.org/licenses/by/4.0/legalcode",
        }
    ],
    "languages": [
        {"id": "eng", "title_l10n": "English", "props": {"alpha_2": "en"}, "tags": ["individual", "living"]}
    ],
    "publisher": "B2SHARE",
}
RDM_SHARE_CITATION={
    "twitter": True,
    "facebook": True,
    "linkedin": True,
    "reddit": False,
    "email": False,
    "mastodon": True,
    "bluesky": True,
}

APP_RDM_DETAIL_SIDE_BAR_TEMPLATES=[
    "invenio_app_rdm/records/details/side_bar/manage_menu.html",
    "invenio_app_rdm/records/details/side_bar/actions.html",
    "invenio_app_rdm/records/details/side_bar/metrics.html",
    "invenio_app_rdm/records/details/side_bar/versions.html",
    "invenio_app_rdm/records/details/side_bar/external_resources.html",
    "invenio_app_rdm/records/details/side_bar/communities.html",
    "invenio_app_rdm/records/details/side_bar/keywords_subjects.html",
    "invenio_app_rdm/records/details/side_bar/details.html",
    "invenio_app_rdm/records/details/side_bar/licenses.html",
    "invenio_app_rdm/records/details/side_bar/citations.html",
    "invenio_app_rdm/records/details/side_bar/export.html",
    "invenio_app_rdm/records/details/side_bar/technical_metadata.html",
]

APP_RDM_RECORD_LANDING_PAGE_TEMPLATE="invenio_app_rdm/records/detail.html"

OAI-PMH configuration#

OAISERVER_ID_PREFIX="b2share-"
OAISERVER_XSL_URL="/static/oai2.xsl"
OAISERVER_ADMIN_EMAILS=None
OAISERVER_METADATA_FORMATS=deepcopy({
    **OAISERVER_METADATA_FORMATS,
    "eudatcore": {
        "serializer": "b3share.resources.serializers.oai:eudatcore_etree",
        "schema": eudatcore_schema,
        "namespace": eudatcore_namespace
    },
    "eudatextended": {
        "serializer": "b3share.resources.serializers.oai:eudatextended_etree",
        "schema": eudatextended_schema,
        "namespace": eudatextended_namespace
    },
})

Storage configuration#

APP_RDM_ARCHIVE_DOWNLOAD_MAX_BYTES=None
FILE_DOWNLOAD_OFFLOAD_ALIAS_ROOT="/opt/invenio/var/instance/data"
FILE_DOWNLOAD_OFFLOAD_ACCEL_PREFIX="/__files_internal"
FILES_REST_XSENDFILE_ENABLED=True
FILES_REST_STORAGE_FACTORY=storage_factory # b3share.file_storage.config.storage_factory
RDM_FILES_DEFAULT_QUOTA_SIZE=20*10**9
RDM_FILES_DEFAULT_MAX_FILE_SIZE=10*10**9
APP_RDM_DEPOSIT_FORM_QUOTA={"maxFiles": 100, "maxStorage": 20 * 10**9}

Feature configuration#

# User profiles
USERS_RESOURCES_ADMINISTRATION_ENABLED=True
USERS_RESOURCES_GROUPS_ENABLED=False
USERPROFILES_READ_ONLY=False
USERPROFILES_EMAIL_ENABLED=False
USERPROFILES_EMAIL_SHOW=True

# Administration (e.g. Audit logs & Jobs)
AUDIT_LOGS_ENABLED=True
JOBS_ADMINISTRATION_ENABLED=True
DEFAULT_ADMIN_USERS=[] # List of emails

# Records, communities and review
CHECKS_ENABLED=True
COMMUNITIES_COLLECTIONS_ENABLED=False
COMMUNITIES_ADMINISTRATION_DISABLED=False
RDM_COMMUNITY_REQUIRED_TO_PUBLISH=True
RDM_COMMUNITY_SUBMISSION_REQUEST_CLS=checks_requests.CommunitySubmission
RDM_COMMUNITY_INCLUSION_REQUEST_CLS=checks_requests.CommunityInclusion
REQUIRE_COMMUNITY_AGREEMENT=["admin", "manager"]
COMMUNITIES_SHOW_ADMIN_STATS=False
COMMUNITIES_DEBUG_ADMIN_STATS=False
CREATE_RECORD_FROM_TEMPLATE_ENABLED=False
SELECTIVE_FILE_DOWNLOAD_ENABLED=False
OWNERSHIP_TRANSFER_ENABLED=True

# Importers
GITHUB_MANUAL_IMPORT_ENABLED=False
GITHUB_INTEGRATION_ENABLED=False
GITHUB_RELEASE_CLASS=B2shareGithubRelease
GITHUB_CITATION_METADATA_SCHEMA=CitationMetadataSchema
GITHUB_CITATION_FILE="CITATION.cff"
IMPORTERS=[]

# B2INST integration
INSTRUMENT_RESOLVER_ENABLED=False # Record landing page
B2INST_INSTRUMENT_SELECTOR_ENABLED=False
B2INST_API_URL="https://b2inst.gwdg.de/api"
B2INST_UI_URL="https://b2inst.gwdg.de"
B2INST_SEARCH_TIMEOUT=10
B2INST_SEARCH_PAGE_SIZE=10
B2INST_FACETS_CACHE_TTL=600
B2INST_COMMUNITIES_CACHE_TTL=3600
B2INST_SUPPORTED_FACET_TYPES={"terms"}
B2INST_FACET_EXCLUDE={"access_status", "file_type", "resource_types"}

# Other features
USER_FAVOURITES_ENABLED=False

Theme configuration#

THEME_LOGO="images/B2SHARE_logo_no_background.png"
THEME_SITENAME="B2SHARE"
THEME_FRONTPAGE_TITLE="Welcome to B2SHARE"
SITE_FOOTER_IMAGE_1="/static/images/eu.png"
SITE_FOOTER_IMAGE_2="/static/images/eosc-hub-g.png"
SITE_FOOTER_TEXT="B2SHARE is co-funded by the EOSC-hub project (Horizon 2020) under Grant number 777536"
SITE_HEADER_TEXT="Store and publish your research data"
SITE_HEADER_PARAGRAPH="Search in public datasets or register as a user to upload and publish your data!"
EUDAT_WEBSITE_LINK="https://eudat.eu"
MENU_HELP_LINK="/help-page"
TOU_LINK="/terms-of-use"
CA_LINK="/community-agreement"
INSTANCE_THEME_FILE="./less/theme.less"

BABEL_DEFAULT_LOCALE="en"
BABEL_DEFAULT_TIMEZONE="Europe/Zurich"
I18N_LANGUAGES=[]

THEME_SHOW_FRONTPAGE_INTRO_SECTION=False

# Static pages
APP_RDM_PAGES={
    "help-page": "/help-page",
    "community-agreement": "/community-agreement",
    "terms-of-use": "/terms-of-use",
}

# Footer links
FOOTER_ABOUT={
    "About EUDAT": {"link": "https://eudat.eu/what-eudat", "external": True},
    "Production site": {"link": "https://b2share.eudat.eu", "external": True},
    "Help": {"link": "/help-page", "external": False},
    "Service Description": {"link": "https://docs.eudat.eu/b2share/service-description/", "external": True},
}
FOOTER_CONTACT={
    "Contact": {"link": "https://eudat.eu/contact-support-request?service=B2SHARE", "external": True},
    "Documentation": {"link": "https://docs.eudat.eu/b2share/overview/", "external": True},
}
FOOTER_POLICYS={
    "Acceptable use policy": {"link": "https://www.eudat.eu/eudat-cdi-aup", "external": True},
    "Data privacy statement": {"link": "https://csc-eudat.pages.eudat.eu/privacy-policy/CDI_files/CDI-B2SHARE_privacy_notice_en/", "external": True},
    "Community agreement": {"link": "/community-agreement", "external": False},
    "Terms of use": {"link": "/terms-of-use", "external": False},
}

# Text area
ALLOWED_HTML_ATTRS={
    "*": ["class"],
    "a": ["href", "title", "name", "class", "rel", "target"],
    "abbr": ["title"],
    "acronym": ["title"],
}

Search configuration#

RDM_FACETS=rdm_facets
RDM_SEARCH=rdm_search
RDM_SEARCH_OPTIONS_CLS=B2SHARESearchOptions
RDM_SEARCH_ARGS_SCHEMA=B2SHARESearchRequestArgsSchema
COMMUNITIES_RECORDS_SEARCH=deepcopy(rdm_search)
EXTENDED_USERS_SORT_OPTIONS={
    "bestmatch": dict(
        title=_("Best match"),
        fields=["_score"],
    ),
    "newest": dict(
        title=_("Newest"),
        fields=["-id"],
    ),
}
EXTENDED_USERS_SEARCH={
    "facets": [],
    "sort": ["bestmatch", "newest"],
}
rdm_facets = {**DEFAULT_RDM_FACETS, **B2SHARE_FACETS}
rdm_search = {
    **DEFAULT_RDM_SEARCH,
    "facets": DEFAULT_RDM_SEARCH["facets"] + list(B2SHARE_FACETS.keys()),
    "query_parser_cls": QueryParser.factory(
        mapping={
            "internal_notes.note": RestrictedTerm(system_permission),
            "internal_notes.id": RestrictedTerm(system_permission),
            "internal_notes.added_by": RestrictedTerm(system_permission),
            "internal_notes.timestamp": RestrictedTerm(system_permission),
            "_exists_": RestrictedTermValue(system_permission, word=word_internal_notes),
        },
        tree_transformer_cls=SearchFieldTransformer,
    ),
}

Serializer configuration#

APP_RDM_RECORD_EXPORTERS={**_APP_RDM_RECORD_EXPORTERS, **B2SHARE_RECORD_EXPORTERS}
RDM_RECORDS_SERIALIZERS={**_record_serializers, **b2share_record_serializers}
b2share_record_serializers={
    "application/eudatcore": ResponseHandler(
        EudatCore11XMLSerializer(), headers=etag_headers
    ),
    "application/vnd.eudat.eudatcore+json": ResponseHandler(
        EudatCore11JSONSerializer(), headers=etag_headers
    ),
    "application/vnd.eudat.eudatcore+xml": ResponseHandler(
        EudatCore11XMLSerializer(), headers=etag_headers
    ),
    "application/eudatextended": ResponseHandler(
        EudatExtended11XMLSerializer(), headers=etag_headers
    ),
    "application/vnd.eudat.eudatextended+json": ResponseHandler(
        EudatExtended11JSONSerializer(), headers=etag_headers
    ),
    "application/vnd.eudat.eudatextended+xml": ResponseHandler(
        EudatExtended11XMLSerializer(), headers=etag_headers
    )
}
B2SHARE_RECORD_EXPORTERS={
    "eudatcore": {
        "name": _("EudatCore"),
        "serializer": (
            "b3share.resources.serializers:EudatCore11XMLSerializer"
        ),
        "params": {},
        "content-type": "application/vnd.eudat.eudatcore+xml",
        "filename": "{id}.xml",
    },
    "eudatcore-json": {
        "name": _("EudatCore JSON"),
        "serializer": (
            "b3share.resources.serializers:EudatCore11JSONSerializer"
        ),
        "params": {"options": {"indent": 2, "sort_keys": True}},
        "content-type": "application/vnd.eudat.eudatcore+json",
        "filename": "{id}.json",
    },
    "eudatcore-xml": {
        "name": _("EudatCore XML"),
        "serializer": (
            "b3share.resources.serializers:EudatCore11XMLSerializer"
        ),
        "params": {},
        "content-type": "application/vnd.eudat.eudatcore+xml",
        "filename": "{id}.xml",
    },
    "eudatextended": {
        "name": _("EudatExtended"),
        "serializer": (
            "b3share.resources.serializers:EudatExtended11XMLSerializer"
        ),
        "params": {},
        "content-type": "application/vnd.eudat.eudatextended+xml",
        "filename": "{id}.xml",
    },
    "eudatextended-json": {
        "name": _("EudatExtended JSON"),
        "serializer": (
            "b3share.resources.serializers:EudatExtended11JSONSerializer"
        ),
        "params": {"options": {"indent": 2, "sort_keys": True}},
        "content-type": "application/vnd.eudat.eudatextended+json",
        "filename": "{id}.json",
    },
    "eudatextended-xml": {
        "name": _("EudatExtended XML"),
        "serializer": (
            "b3share.resources.serializers:EudatExtended11XMLSerializer"
        ),
        "params": {},
        "content-type": "application/vnd.eudat.eudatextended+xml",
        "filename": "{id}.xml",
    }
}

Service configuration#

RDM_PERMISSION_POLICY=B3ShareRDMRecordPermissionPolicy
RDM_FILE_RESOURCE_CLS=B2ShareFileResource
COMMUNITIES_MEMBERS_SERVICE_COMPONENTS=[
    MetadataComponent,
    CommunityMemberCachingComponent,
    MembershipStatsComponent,
]

Statistics configuration#

STATS_QUERIES=STATS_QUERIES
STATS_QUERIES={
    **_upstream_stats_queries,
    "community-record-view": {
        "cls": TermsQuery,
        "permission_factory": None,
        "params": {
            "index": "stats-record-view",
            "doc_type": "record-view-day-aggregation",
            "copy_fields": {
                "communities": "communities",
            },
            "query_modifiers": [],
            "required_filters": {
                "communities": "communities",
            },
            "metric_fields": {
                "views": ("sum", "count", {}),
                "unique_views": ("sum", "unique_count", {}),
            },
        },
    },
    "community-record-download": {
        "cls": TermsQuery,
        "permission_factory": None,
        "params": {
            "index": "stats-file-download",
            "doc_type": "file-download-day-aggregation",
            "copy_fields": {
                "communities": "communities",
            },
            "query_modifiers": [],
            "required_filters": {
                "communities": "communities",
            },
            "metric_fields": {
                "downloads": ("sum", "count", {}),
                "unique_downloads": ("sum", "unique_count", {}),
                "data_volume": ("sum", "volume", {}),
            },
        },
    },
}