Welcome back to Part 2 of OCI Functions. If you didn’t catch Part 1 you can read it here. In this part, we are going to take a look at creating our own OCI Function in Python. We know from working with the Pre-Built Function we have some things to think about.

• What does our function do?
• What network connectivity will it need?
• What permissions will it need within OCI?
• Do we need some tags for reporting and cost management?
• Where will we develop the function?

Plan

Let us set out a plan for our function.

Function Task

We’ll use our function to grab some data from the Office of National Statistics. In fact, we will collect the UK population by year using this URL:

https://www.ons.gov.uk/generator?format=csv&uri=/peoplepopulationandcommunity/populationandmigration/populationestimates/timeseries/ukpop/pop

We will take the data as is and make it available in OCI as an object in a storage bucket.

Connectivity

We do not need ingress to the function only egress so our VCN needs to have internet access and we can connect our application to the private subnets. Lucky for us in Part 1 created a VCN that matches this exact connectivity requirement!

Permissions

The function will need to be able to create objects in a storage bucket.

Reporting

For now, we won’t consider using tags or how will report usage, if there is enough interest we might cover that in a later blog post.

Development Environment

I don’t know about you, but I really don’t want to set up a development environment! We are indeed blessed, Oracle has provided a Cloud Code Editor and we are going to make use of it today and the superb integration it offers with OCI.

Code Editor

When you log into OCI towards the top right we see a Developer Tools icon and from there we can select Code Editor.

This editor is really good, supports lots of languages and benefits from integration with OCI. I love the Terminal that can be embedded within the Editor combining the elegance of the Cloud Shell and the richness of a functional IDE.

Creating a Folder

I’m going to use the Terminal to create the folder that we will be working in. I’ll opt for the Terminal in most cases simply because it's easier to share in the blog for you to follow along.

mkdir oci-functions-part-2

In this blog, we will reuse the Application that was created in Part 1. It has the network connectivity we need and saves us from repeating the same information here.

Setup Fn

Again using the Terminal we can set up our Fn CLI. List the available contexts and select the one you want to use. These next few commands are available under your Application on the Getting Started page.

fn list context
fn use context uk-london-1

Now we need to update oracle.compartment-id, the easiest way to get the compartment id is to navigate using the OCI console or simply copy the command from the Getting Started page under your application.

fn update context oracle.compartment-id ocid1.compartment.oc1..aaaa....

The next step is to update Fn with the registry that will be used to store the image that is created for our function.

fn update context registry [region-key].ocir.io/[tenancy-namespace]/[repo-name-prefix]

Now you can set up an Auth Token under your user profile. This is basically an App Password so you don’t have to share your actual password with services. If you don’t have an Auth Token to use set one up and save the token for future use.

Now you can login using:

docker login -u '[tenancy-namespace]/oracleidentitycloudservice/[username]' [region-key].ocir.io

Creating the Function

Whilst in the Terminal pane we can set up the function.

fn init --runtime python get-population-data

Using the Code Editor we can now open the containing folder called oci-functions-part-2.

But what is this we see, an error in the boilerplate?! It's just a case of the fdk library not being installed. In the Terminal pane we can sort this out by running:

pip install fdk --user

You might need to reopen the editor for the change to be recognised.

Updating func.py

Here is the code that we will use for the function definition.

import io
import json
import requests
import oci
import logging
import fdk

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def handle(ctx, payload):
    # Extract the URL and bucket details from the payload
    try:
        body = json.loads(payload)
        url = body["url"]
        bucket_name = body["bucket_name"]
        object_name = body["object_name"]
    except (ValueError, KeyError) as e:
        logger.error(f"Invalid payload or missing parameters: {str(e)}")
        return {"error": "Invalid payload or missing parameters"}, 400  # 400 Bad Request

    # Download the CSV file
    try:
        response = requests.get(url)
        response.raise_for_status()
        content = response.content
    except requests.RequestException as e:
        logger.error(f"Error downloading file from URL: {str(e)}")
        return {"error": str(e)}, 500  # 500 Internal Server Error

    # Upload the content to OCI Object Storage
    try:

        signer = oci.auth.signers.get_resource_principals_signer()
        object_storage = oci.object_storage.ObjectStorageClient({}, signer=signer)

        namespace = object_storage.get_namespace().data

        put_response = object_storage.put_object(
            namespace, 
            bucket_name, 
            object_name, 
            io.BytesIO(content)
        )

        if put_response.status != 200:
            logger.error("Failed to upload to OCI bucket")
            return {"error": "Failed to upload to OCI bucket"}, 500  # 500 Internal Server Error

    except oci.exceptions.ServiceError as e:
        logger.error(f"Error uploading to OCI bucket: {str(e)}")
        return {"error": str(e)}, 500  # 500 Internal Server Error

    logger.info("CSV file downloaded and uploaded to OCI successfully!")
    return {"success": True, "message": "CSV file downloaded and uploaded to OCI successfully!"}, 200  # 200 OK

def main(ctx, data: io.BytesIO = None):
    payload = data.getvalue().decode('utf-8')
    response_body, status_code = handle(ctx, payload)
    return fdk.response.Response(
        ctx,
        response_data=json.dumps(response_body),
        status_code=status_code,
        headers={"Content-Type": "application/json"}
    )

Our function expects a JSON payload containing the parameters.

Updating func.yaml

We’re show boating a little bit, instead of using the expected handler function name, we used another function name called main. So we need to update the yaml file to reflect that usage.

schema_version: 20180708
name: get-population-data
version: 0.0.3
runtime: python
build_image: fnproject/python:3.9-dev
run_image: fnproject/python:3.9
entrypoint: /python/bin/fdk /function/func.py main
memory: 256

As you can see on line 7 we have changed the function name to main. You can use this technique to use functions from different files.

Updating requirements.txt

Finally, we make a change to requirements.txt, in this case, we are using some libraries that are not shipped with fn so we add them here.

fdk>=0.1.60
requests==2.31.0
oci==2.112.0

Deploying the Function

This is super simple from within the Code Editor. We open the terminal and change to the function directory. Then run the deploy command:

fn -v deploy --app data-engineering-oci-functions-application

If the function was deployed successfully we eventually get confirmation in the terminal.

Updating function get-population-data using image [region-code].ocir.io/[tenancy-namespace]/data-engineering-part-two/get-population-data:0.0.4...
Successfully created function: get-population-data with [region-code].ocir.io/[tenancy-namespace]/data-engineering-part-two/get-population-data:0.0.4

We can test the deployment with our knowledge from Part 1. To test our function we need a payload.

{
    "url": "https://www.ons.gov.uk/generator?format=csv&uri=/peoplepopulationandcommunity/populationandmigration/populationestimates/timeseries/ukpop/pop",
    "bucket_name": "data-engineering-oci-functions-destination",
    "object_name": "ons-population-data.csv"
}

Then we can invoke the function with the payload.

cat get-population-data-payload.json | fn invoke data-engineering-oci-functions-application get-population-data

Well, at least we know the function has been deployed!

{"error": "{'target_service': 'object_storage', 'status': 404, 'code': 'BucketNotFound', 'opc-request-id': '...', 'message': \"Either the bucket named 'data-engineering-oci-functions-destination' does not exist in the namespace '...' or you are not authorized to access it\", 'operation_name': 'put_object', 'timestamp': '2023-09-10T15:33:30.153337+00:00', 'client_version': 'Oracle-PythonSDK/2.112.0', 'request_endpoint': 'PUT https://objectstorage.uk-london-1.oraclecloud.com/n/.../b/data-engineering-oci-functions-destination/o/ons-population-data.csv', 'logging_tips': 'To get more info on the failing request, refer to https://docs.oracle.com/en-us/iaas/tools/python/latest/logging.html for ways to log the request/response details.', 'troubleshooting_tips': \"See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_bucketnotfound for more information about resolving this error. Also see https://docs.oracle.com/iaas/api/#/en/objectstorage/20160918/Object/PutObject for details on this operation's requirements. If you are unable to resolve this object_storage issue, please contact Oracle support and provide them this full error message.\"}"}

At this point, we need to give the function some permissions to operate on the bucket.

Permissions

Let's start by creating a dynamic group. Give it a name, I’ve used data-engineering-functions-get-population-data and a description. Then add a rule, I’m keeping this simple and assigning the function to the group using its OCID.

All {resource.id = 'ocid1.fnfunc.oc1.uk-london-1.aaa....'}

Now we should create a policy that gives the dynamic group permission to store objects in the bucket.

Allow dynamic-group data-engineering-functions-get-population-data to read buckets in compartment data-engineering-oci-functions
Allow dynamic-group data-engineering-functions-get-population-data to manage objects in compartment data-engineering-oci-functions
Allow dynamic-group data-engineering-functions-get-population-data to read objectstorage-namespaces in compartment data-engineering-oci-functions
Allow dynamic-group data-engineering-functions-get-population-data to read compartments in compartment data-engineering-oci-functions

Now if we test our function we get success messages!

{"success": true, "message": "CSV file downloaded and uploaded to OCI successfully!"}

In Summary

Oracle provides us with a rich set of tools right there in the OCI environment:

• Code Editor to edit our code files using an IDE
• Cloud Shell to interact with the Fn Server
• OCI Console to navigate the different services and logs

We can use these tools to create Functions in the cloud with no need to configure any desktop machines. When creating a function we need to think about:

• How the Functions will be grouped into Applications
• The network connectivity the Application needs and how we should configure a VCN
• How our Functions share similar permissions and whether they can be contained in Dynamic Groups
• How the permissions can be described as Statements and defined within Policies

In the next post, we will take a closer look at Dynamic Groups, Policies and the Logs within the context of Function development. If you want to be informed of when that blog is posted please subscribe!