The other day I wrote about my little Drag and Drop application and mentioned I wanted it to send HTTPS POST requests to an existing PHP web application. I thought it would be a relatively trivial task, but it turned out not the be as easy as I thought. First let me show you the code, and then discuss the pitfalls.
Sending the request by itself is actually pretty easy. Observe:
// this is what we are sending
string post_data = "foo=bar&baz=oof";
// this is where we will send it
string uri = "https://someplace.example.com";
// create a request
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create(uri); request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
request.Method = "POST";
// turn our request string into a byte stream
byte[] postBytes = Encoding.ASCII.GetBytes(str);
// this is important - make sure you specify type this way
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postBytes.Length;
Stream requestStream = request.GetRequestStream();
// now send it
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
// grab te response and print it out to the console along with the status code
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
Console.WriteLine(response.StatusCode);
Piece of cake, right? Even if you don’t know C# you should be able to follow this. As a side note, can you see how freakishly verbose this shit really is? I mean, how come ruby can implement this like this:
Net::HTTP.post_form(URI.parse('https://somplace.example.com'),
{'foo'=>'bar', 'bas'=>'off'})
Or python for that matter:
data = urllib.urlencode({"foo" : "bar", "baz" : "oof"})
urllib.urlopen("https://somplace.example.com", data)
Is it really necessary for me to declare all that shit and go through all the motions to perform something that other languages do in a single line? This is one of these reasons why I no longer think that the traditional Java/C# approach of static typing, and overwhelming verbosity is the way to go. The functionality here is the same – and yet Ruby and Python totally win on expressiveness and readability hands down.
But I didn’t really start this post (just) to rag on the verbosity of C#. The problem with the code above is that it doesn’t work if your certificate is not valid. Why would I be posting to a web page with and invalid SSL certificate? Because I’m cheep and I didn’t feel like paying Verisign or one of the other jerk-offs for a cert to my test box so I self signed it. When I sent the request I got a lovely exception thrown at me:
System.Net.WebException The underlying connection was closed. Could not establish trust relationship with remote server.
I don’t know about you, but to me that exception looked like something that would be caused by a silly mistake in my code that was causing the POST to fail. So I kept searching, and tweaking and doing all kinds of weird things. Only after I googled the damn thing I found out that the default behavior after encountering an invalid SSL cert is to throw this very exception.
Fortunately, there is a quick and dirty workaround. You simply need to subclass the ICertificatePolicy class from the System.Security.Cryptography.X509Certificates namespace and rig it so that it always validates the certificate, no matter what:
using System.Security.Cryptography.X509Certificates;
using System.Net;
public class MyPolicy : ICertificatePolicy
{
public bool CheckValidationResult(ServicePoint srvPoint,
X509Certificate certificate, WebRequest request,
int certificateProblem)
{
//Return True to force the certificate to be accepted.
return true;
}
}
Once you do that, you just need to toss the following line somewhere in your code, before you actually send the request. This will swap your default CertificatePolicy class for our fake one with the validation hack:
System.Net.ServicePointManager.CertificatePolicy
= new MyPolicy();
Note that the compiler may whine about this approach being an obsolete, but since this is a pretty ugly hack in itself, I paid no heed to it. There might be a better way to do this in .NET 3.5 but since it worked I let it be for now.
So there is how you do it. A more relevant exception message would be a lot of help in this circumstance. Perhaps something among the lines of “invalid certificate”? But I’m just thinking out loud here. If you ever run into this issue, here is how to solve it.
[tags]c#, .net, https, http, post, https post, http post[/tags]
Great article, specially the certificate section.
I have a problem. The variable post_data, the one you say you will be sending, is not used anywhere in the code you provide.
Thank you,
GG
@George Gomez: Oh, yeah – copy and paste blunder.
Should be:
I hope this clears things up.
Yes it does, thank you very much.
Very Good!!
I tried to use the code posted in http://geekswithblogs.net/rakker/archive/2006/04/21/76044.aspx, it’s good… but it fails when i need to connect to an HTTPS. So your article was perfect to solve that problem.
Thanks!
I completely agree with your assessment that the design of the API is quite crap because it makes you write so much unnecessarily verbose shit.
However, you are wrong in claiming that this is due to static typing. It is entirely a problem of .NET’s API. Even without sacrificing static typing, one could easily have a static class Http with a static method Http.Post() that allows you to do this in a single line of code, for example:
Http.Post(“https://somplace.example.com”, new Dictionary() { { “foo”, “bar” }, { “baz”, “oof” } });
I notice from your posted code that Ruby requires a completely superfluous call to URI.parse(). Python requires you to call urlencode() and then handle the encoded data, which is bad practice (incomplete encapsulation). Unfortunately .NET is even worse, but it shows that everyone makes mistakes in designing their APIs.
@Timwi: Yeah, you are right. It’s the API that’s the issue here, not static typing.
Although, I must admit that the two often go together – I mean static typing, and overly verbose API’s. Or maybe it’s just that C# was designed to look and feel like Java and so it inherited it’s verbosity.
I disagree. The WebRequest API is the only example I can think of where C# is so overly verbose. Except for that, I feel that C# has precisely not copied Java’s verbosity. I can’t think of any other examples that are grossly more verbose than necessary.
Of course C# is still slightly more verbose than dynamically-typed languages like Python, but clearly not unnecessarily so: the static typing you get saves you more time in frustrating debugging than a few characters can save you in typing.
Just a quick note: automatically accepting a self-signed SSL cert is bad security if there is anything sensitive in your POST. Someday there could be a man-in-the-middle reading your POST and you wouldn’t even know it.
As for excessively verbose API, Java’s regex API is like this too. Compile a string into a Pattern. Use the Pattern to make a Matcher. Then call various methods on the Matcher and check the results. When it comes to regex, Ruby and Perl do in 1 line what Java does in 6.
I ran into this exact thing with Java and the solution is pretty much the same there: fool the underlying security class into accepting all certs. X.509 is pretty well thought out, but really difficult to implement correctly. Getting all parties to agree on the web of trust is hard enough in a closed environment like a lab, but sucks in large corps. Thanks for the post; I’m climbing the C# learning curve.
Dude,
I love this article. I laughed I concurred. It was definitely better than Cats!
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
From MSDN: “ReadToEnd assumes that the stream knows when it has reached an end. For interactive protocols, in which the server sends data only when you ask for it and does not close the connection, ReadToEnd might block indefinitely and should be avoided.”
TCP considers the “End” as the closing of the TCP connection, since it has no other way of knowing if more data is on the way. If the server never closes the TCP connection, your call will bock until it does. You should use the read function to read in the number of bytes specified by the ContentLength Header of the HTTPRequest. This way you are not dependent upon the closing of the TCP connection from the server side to continue.
What i really want to know, how can I send some parameters, and these parameters enter into a field of type “textbox” then I would take the request of this consultation and analyze the data.
I can do the sample code but is doesn’t solution my problem.
how I do that?
sorry english.
@ Leandro:
It’s easy – just append all the parameters into a string like this:
The format is “argument1=value1&argument2=value2” and etc…
Does that make sense?
@ Luke Maciak:
I can do that, but just in pages that submit button that send me to a another page.
The problem that i find now is:
The event button of the site that I need to send data, is an funcion of javascript. Is it possible?
Thank You again
@ Leandro:
I’m not entirely sure what you mean here. The code above will send a POST request to a URL of your choice. This is equivalent of a submitting a form. For example, when you hit a submit button on a HTML form it usually sends either POST or GET request to some other page (or to itself).
These request must then be handled on the server side. If you are trying to build an interface for an existing web form you should probably send the POST request to wherever that form is going. Look at the HTML code for your form. It should be something like this:
The action attribute tells you where to send the POST request.
Now if the form has some sort of client-side javascript check going on, you probably have to recreate that behavior in your code. A lot of that stuff is session related or anti-spam protection so there is really no clear cut answer here.
I guess my question is – do you have access to the server or are you building this tool for an existing website? If you do have access to the server, you can probably create a web service type page for yourself which will accept POST requests and handle them appropriately.
Pingback: Collecting Hardware Information using C# and WMI « Terminally Incoherent
really nice article!
i have a problem while using ‘&’ , ‘=’ symbols in post_data as a value. tried
using quotes.
ex: if i want send x&yz as one of the value in post_data i am able to receive only x. the y after & is interpreted as another parameter…
@ Gupta:
Yes, this is by design. The server sees all the GET and POST requests as a single string like this:
From there it parses it to extract the necessary value pairs. If you want to send a vale that contains characters & or = you need to URL encode them:
This will convert the special characters into html entities. When you need to retrieve the value back just run the corresponding UrlDecode over it to change it back.
I hope this helps.
Just thought you’d like to know, I’ve looked this up to implement it for one of your sponsors.
How good is that.
Great post BTW
is there any way to get the encoding type of the site.. for example in this sentance
byte[] postBytes = Encoding.ASCII.GetBytes(str);
you have ASCII encoding. so, how would i know if it is ASCII or UTF or other encoding…
need help
thanks… good guide
does anyone knows the error “premature end of file” after sending an HTTPS Post as described?
Careful refactoring… If you set the POST method after you populate the postBytes RequestStream, it compiles, but gives you the exception
If you set it first, then set
request.Method = "POST"again later, it wipes your data.does anyone knows the error “premature end of file” after sending an HTTPS Post as described?
Any insight on the following?
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.DllNotFoundException: Unable to load DLL ‘security.dll’: The specified module could not be found. (Exception from HRESULT: 0x8007007E) at System.Net.UnsafeNclNativeMethods.SafeNetHandles_SECURITY.EnumerateSec urityPackagesW(Int32& pkgnum, SafeFreeContextBuffer_SECURITY& handle) at System.Net.SafeFreeContextBuffer.EnumeratePackages(SecurDll Dll, Int32& pkgnum, SafeFreeContextBuffer& pkgArray) at System.Net.SSPISecureChannelType.EnumerateSecurityPackages(Int32& pkgnum, SafeFreeContextBuffer& pkgArray) at System.Net.SSPIWrapper.EnumerateSecurityPackages(SSPIInterface SecModule) at System.Net.SSPIWrapper.GetVerifyPackageInfo(SSPIInterface secModule, String packageName, Boolean throwIfMissing) at System.Net.Security.SecureChannel..ctor(String hostname, Boolean serverMode, SchProtocols protocolFlags, X509Certificate serverCertificate, X509CertificateCollection clientCertificates, Boolean remoteCertRequired, Boolean checkCertName, Boolean checkCertRevocationStatus, LocalCertSelectionCallback certSelectionDelegate) at System.Net.Security.SslState.ValidateCreateContext(Boolean isServer, String targetHost, SslProtocols enabledSslProtocols, X509Certificate serverCertificate, X509CertificateCollection clientCertificates, Boolean remoteCertRequired, Boolean checkCertRevocationStatus, Boolean checkCertName) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.ConnectStream.WriteHeaders(Boolean async) — End of inner exception stack trace — at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at kerberosTest._Default.Page_Load(Object sender, EventArgs e)
does anyone knows the error “premature end of file” after sending an HTTPS Post as described?
@ Luke Maciak:
Hi
Great post! Thanks much.
I have a doubt, it there a way to POST a file to an URI, instead of a string (in this example post_data)
Hi Luke,
Thanks for a great post, really helped me out (and made me laugh along the way).
You might wish to attach a CookieContainer to the request and pass it onward to further requests on that domain.
CookieContainer cookieJar = new CookieContainer();
request.CookieContainer = cookieJar;
.
.
.
HttpWebRequest secondRequest = (HttpWebRequest)WebRequest.Create(ANOTHER_URI);
secondRequest.CookieContainer = cookieJar;
Hi, could you tell me how it will look like the PHP script to process the request and response?
Sorry for my English:)
@ Jiri Zapletal:
Really dude? It’s a regulat POST request. If you are using PHP then you do something like:
I mean, that’s all there is to it. The code above just sends a POST request and passes along the values of foo and bar.
It didn’t work for me.. I got below mentioned error. when i try to access https site from my app server.
From my app server port 80 is blocked in firewall. only port 443 is opened
Unable to connect to the remote server Exception : System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 141.13.20.119:8080 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
@ Jason:
You are missing a DLL.. It says right at the beginning that it can’t find a required DLL.
Hi luke
Tahnks for the article.. but i had a secure connection site,which accept the data only if the URL contain the username and password:
eg:
https://username:password@ip.com?post_data
and it is giving me the error :
The remote server returned an error: (401) Unauthorized.
on line:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
please assist me….
Hello, I think your blog might be having browser compatibility issues.
When I look at your blog in Safari, it looks fine but when
opening in Internet Explorer, it has some overlapping.
I just wanted to give you a quick heads up! Other then that, wonderful blog!
New code to do the same thing, without using the obsolete CertificatePolicy property –
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) { return true; };
Hi. Nice blog. It really helped me out .
But I am facing an issue on POST -ing something to the SSL enabled server.
Error message 401.3: You do not have permission to view this directory or page using the credentials you supplied.
While debugging, when the POST is sent, your code is executed and it returns the ‘true’ value. On coming back to POST, it throws an exception as shown above. Any inputs?
Thank You!
BenN wrote:
New code to do the same thing, without using the obsolete CertificatePolicy property –
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) { return true; };This works perfectly. No MyPolicy class required. Thanks
Came for the code, leave saying that I agree that c# is waaaaaay to verbose compared to other languages.
hey
I want to be logged in to wigle.net the help of your code, but it does not work for me on this page, you may know why.
Pingback: How to: Could not establish trust relationship for SSL/TLS secure channel — SOAP | SevenNet
Pingback: Fix: Could not establish trust relationship for SSL/TLS secure channel — SOAP #dev #development #solution | Good Answer
@ Luke Maciak:
That whole line displays an appaling lack of text encoding awareness, actually. You use ascii, of all things; that strips out any nonstandard characters. Not even utf-8, the de facto standard, or ISO-8859-1, which is known for preserving the actual bytes no matter what.
See The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)