Need insights of help in creating or connecting Jupyter with AWS EMR serverless APPLICATION using Livy endpoint (original) (raw)

Hello Guys

I am trying a POC which is connecting my Jupyterhub installed in Amazon Ec2 box (pip install ) with AWS EMR serverless Application via Livy endpoint point

Method 1 : Creating inetarctive sessions using sparkmagic from jupyter notebook is working when we do with jupyter user

Method 2: In this method we are joining the Ec2 with ADJoin as the server is adjoined post that we are not able to have a interactive sessions from jpuyter box to EMR serverless via Livy endpoint

example when we run below quesry in jupyter notebook we get an error mentioned below

%spark
%info

Error message:

An error was encountered:
You need to have at least 1 client created to execute commands.

query 2:
%spark add -s my_session -l python -u https://.livy.emr-serverless-services.us-east-1.amazonaws.com

Anvalid status count 4eg" from
with error payload: {“message”:“Missing Authentication Token”}

How to make sure the interactive sessions created Even if the server is adjoined

if any more specific configuration details required i can share here please do let me know your inputs to chase this integration part

image

Thanks
Rahul