HTTP basic authentication in the URL is not available

[lotcher]

I am a novice in network protocol. In other programming languages (such as Python or Julia), I can send a request with authentication using code similar to the following

requests.get('http://elastic:elastic@localhost:9200')  # Python
> <Response [200]>

HTTP.get('http://elastic:elastic@localhost:9200')  # Julia
>  <Response [200]>

But when I use scalaj-http, I get a 401 response

Http('http://elastic:elastic@localhost:9200').asString  //Scala
> <Response [401]>

I've tried other Scala HTTP libraries, such as requests-scala, and the same problem. I want to know http://user:password@address:port is a common way to write? Why doesn't our library support it?

[fntz]

did you use single-quote?
because I got normal response:

Http("http://elastic:elastic@localhost:9200").asString
val res0: scalaj.http.HttpResponse[String] =
HttpResponse({
  "name" : "2057040aa337",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "_hRP22GdTuy9Y1UfH2VeLg",
  "version" : {
    "number" : "7.14.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6bc13727ce758c0e943c3c21653b3da82f627f75",
    "build_date" : "2021-09-15T10:18:09.722761972Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
,200,TreeMap(content-encoding -> Vector(gzip), content-length -> Vector(330), content-type -> Vector(application/json; charset=UTF-8), Status -> Vector(HTTP/1.1 200 OK)...

[lotcher]

Sorry, the quotation marks in the above description are used incorrectly. In the process of my actual request, I explicitly returned the result of 401. The response of ES was authentication failure, which seemed to ignore the account and password in the URL(I don't know if your local ES has authentication?).

Http("http://elastic:elastic@localhost:9200").asString

res120: scalaj.http.HttpResponse[String] = HttpResponse(
  body = "{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}",
  code = 401,
  headers = TreeMap(
    "content-encoding" -> Vector("gzip"),
    "content-length" -> Vector("192"),
    "content-type" -> Vector("application/json; charset=UTF-8"),
    "Status" -> Vector("HTTP/1.1 401 Unauthorized"),
    "WWW-Authenticate" -> Vector("Basic realm=\"security\" charset=\"UTF-8\"")
  )
)

I use auth to get the correct return value.

Http("http://localhost:9200").auth("elastic", "elastic").asString

At the same time, I also found another problem that seems to be related to the above. When there is an '@' character in the password in the URL(In fact, '@' is a legally named character in a password), a [ConnectException: Connection refused] will be thrown, even if the correct account and password are used after auth

Http("http://elastic:elastic@123@localhost:9200/").auth("elastic", "elastic@123").asString

java.net.ConnectException: Connection refused (Connection refused)
  java.net.PlainSocketImpl.socketConnect(Native Method)
  java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  java.net.Socket.connect(Socket.java:607)
  sun.net.NetworkClient.doConnect(NetworkClient.java:175)
  sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
  sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
  sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
  sun.net.www.http.HttpClient.New(HttpClient.java:339)
  sun.net.www.http.HttpClient.New(HttpClient.java:357)
  sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
  sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
  sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
  sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
  sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
  sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
  java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
  scalaj.http.HttpRequest.doConnection(Http.scala:367)
  scalaj.http.HttpRequest.exec(Http.scala:343)
  scalaj.http.HttpRequest.asString(Http.scala:492)
  ammonite.$sess.cmd2$Helper.<init>(cmd2.sc:3)
  ammonite.$sess.cmd2$.<clinit>(cmd2.sc:7)

[fntz]

hm, how did you run elasticsearch ?

I ran with the next command:

docker run -p 9200:9200 -p 9301:9301 --name elastic -e "discovery.type=single-node" -e "xpack.security.enabled=true" -e "ELASTIC_PASSWORD=elastic" -e "ELASTIC_PASSWORD=elastic" elasticsearch:7.14.2

user/password=elastic/elastic

and everything is ok.
I mean:

// correct creds:
 Http("http://localhost:9200").auth("elastic", "elastic").asString
val res2: scalaj.http.HttpResponse[String] =
HttpResponse({
  "name" : "87c02b02bd15",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "EJ0i7OHIS8-8UUP1HAqUjw",
  "version" : {
    "number" : "7.14.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6bc13727ce758c0e943c3c21653b3da82f627f75",
    "build_date" : "2021-09-15T10:18:09.722761972Z",

// incorrect creds:
Http("http://localhost:9200").auth("elastic", "elasti1c").asString
val res3: scalaj.http.HttpResponse[String] = HttpResponse({"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" 

[lotcher]

I tried the same startup method as you

docker run -p 9200:9200 -p 9301:9301 --name elastic -e "discovery.type=single-node" -e "xpack.security.enabled=true" -e "ELASTIC_PASSWORD=elastic" -e "ELASTIC_PASSWORD=test" elasticsearch:7.14.2

To prevent other factors, I run directly in the scala shell

Welcome to Scala 2.13.6 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_231).
Type in expressions for evaluation. Or try :help.

scala> :require /Users/bowaer/Desktop/scalaj-http_2.13-2.4.2.jar
Added '/Users/bowaer/Desktop/scalaj-http_2.13-2.4.2.jar' to classpath.

scala> import scalaj.http._
import scalaj.http._
//If we do not use the account password, we are expected to return 401 directly
scala> Http("http://localhost:9200").asString
val res2: scalaj.http.HttpResponse[String] = HttpResponse({"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401},401,TreeMap(content-encoding -> Vector(gzip), content-length -> Vector(192), content-type -> Vector(application/json; charset=UTF-8), Status -> Vector(HTTP/1.1 401 Unauthorized), WWW-Authenticate -> Vector(Basic realm="security" charset="UTF-8")))

//Similarly, when using .auth(), we can also return the correct results
scala> Http("http://localhost:9200").auth("elastic", "test").asString
val res4: scalaj.http.HttpResponse[String] =
HttpResponse({
  "name" : "b0d0c92838d1",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "t76LLMsYThyoaFY0y0HyXQ",
  "version" : {
    "number" : "7.14.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6bc13727ce758c0e943c3c21653b3da82f627f75",
    "build_date" : "2021-09-15T10:18:09.722761972Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
,200,TreeMap(content-encoding -> Vector(gzip), content-length -> Vector(332), content-type -> Vector(application/json; charset=UTF-8), Status -> Vector(HTTP/1.1 200 OK)...

//However, when we put the account password in the URL, we still return 401
scala> Http("http://elastic:test@localhost:9200").asString
val res5: scalaj.http.HttpResponse[String] = HttpResponse({"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401},401,TreeMap(content-encoding -> Vector(gzip), content-length -> Vector(192), content-type -> Vector(application/json; charset=UTF-8), Status -> Vector(HTTP/1.1 401 Unauthorized), WWW-Authenticate -> Vector(Basic realm="security" charset="UTF-8")))

//Similarly, when we use .auth() and URL authentication at the same time, we can still get the right return
scala> Http("http://elastic:test@localhost:9200").auth("elastic", "test").asString
val res6: scalaj.http.HttpResponse[String] =
HttpResponse({
  "name" : "b0d0c92838d1",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "t76LLMsYThyoaFY0y0HyXQ",
  "version" : {
    "number" : "7.14.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6bc13727ce758c0e943c3c21653b3da82f627f75",
    "build_date" : "2021-09-15T10:18:09.722761972Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
,200,TreeMap(content-encoding -> Vector(gzip), content-length -> Vector(332), content-type -> Vector(application/json; charset=UTF-8), Status -> Vector(HTTP/1.1 200 OK)...

Sorry, another @ character problem does not reappear in this environment. It may be related to a certain version and environment. I need more tests

[fntz]

you are right, without directly calling the .auth method scalaj do not parse and send the Authorization header.

as a solution, just call .auth or make a wrapper.